Scrape a URL list

A URL scrape collects only the product pages that you send. Use it for two tasks:

• Update a catalog. Refresh the prices and the availability of some products often, and refresh the full catalog less often.
• Scrape only some products. Get the products that you need, without the full catalog.

Before a URL scrape, the website must be in your project. See Add a website.

VariantsCopy Anchor Link

The API selects the variant of each URL scrape. variant on the job shows it.

A full catalog is fresh for 31 days after the start of its full scrape. It is also fresh while a full scrape of your project runs. With a fresh catalog, the variant is always CATALOG_URLS, and urls_only has no effect.

1. Create the URL scrapeCopy Anchor Link

Send POST /jobs/scraping with the list in urls. One job accepts from 1 to 100,000 URLs.

curl https://api.deepscout.ai/v1/jobs/scraping \
  -H "Authorization: Bearer $SCOUT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
    "website_id": "example-shop.cz",
    "urls": [
      "https://www.example-shop.cz/p/bio-kokosovy-olej-500-ml",
      "https://www.example-shop.cz/p/vitamin-c-1000",
      "https://www.example-shop.cz/p/vitamin-c-1000",
      "https://www.other-shop.cz/p/vitamin-d3"
    ]
  }'

The API cleans the list before it creates the job:

• It adds https:// to a URL without a scheme, changes http:// to https:// and changes the domain to lower case.
• It removes URLs that are not valid (invalid).
• It removes URLs of other websites (other_website). The www. and the bare form of the domain are both accepted. Usually, other subdomains are not.
• It removes duplicates (duplicate) and keeps the first copy.

rejected counts the removed URLs for each reason. rejected_urls shows the first 100 invalid and other-website URLs. The job contains only the accepted URLs, and the quote is for these URLs.

{
  "job_id": "jobreq_01k7d5s6t7v8w9x0y1z2a3b4c5",
  "job_type": "URL_SCRAPE",
  "status": "AWAITING_APPROVAL",
  "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
  "website_id": "example-shop.cz",
  "variant": "CATALOG_URLS",
  "url_count": 2,
  "estimate": {
    "currency": "EUR",
    "unit_price_per_1000": "0.70",
    "quantity": 2,
    "total": "0.10",
    "cap": "0.15",
    "min_charge": "0.10",
    "difficulty": "MEDIUM"
  },
  "rejected": { "invalid": 0, "other_website": 1, "duplicate": 1 },
  "rejected_urls": ["https://www.other-shop.cz/p/vitamin-d3"],
  "created_at": "2026-10-13T12:00:00.000Z",
  "updated_at": "2026-10-13T12:00:00.000Z"
}

If the API accepts no URL, it answers 400 INVALID_URL_LIST. An empty list, or a list of more than 100,000 URLs, gets 422. For a larger list, split it into more jobs.

Each request creates a new job, also for the same list.

2. Approve the quoteCopy Anchor Link

Approve the job with POST /jobs/scraping/{job_id}/approve. The approval works as for a full scrape: it holds the cap, and it can answer ESTIMATE_CHANGED, INSUFFICIENT_FUNDS or MONTHLY_LIMIT_REACHED.

The approval also selects the variant again. If your catalog became too old since the quote, the approval answers 409 CATALOG_EXPIRED. If a full scrape made a fresh catalog since the quote, a URLS_ONLY job changes to CATALOG_URLS.

To remove a quote, send POST /jobs/{job_id}/cancel. A quote that you do not approve for 7 days changes to CANCELLED.

When the catalog is too oldCopy Anchor Link

If the full catalog of your project is older than 31 days, the create request or the approval answers 409. This does not occur if the last completed scrape of the website was URLS_ONLY.

{
  "code": "CATALOG_EXPIRED",
  "detail": "the FULL catalog of website example-shop.cz is older than 31 days"
}

Select one of two options:

3. Follow the progress and get the dataCopy Anchor Link

A URL scrape has the same statuses and the same scrape_progress as a full scrape. While it collects pages, percent compares pages with url_count.

To list the URL scrapes of a website, send GET /jobs/scraping with project_id and website_id. The list shows url_count. The list of URLs is only in GET /jobs/scraping/{job_id}.

URL scrapes do not change the state of the website. The job of the website stays the setup job or the newest full scrape.

The charge is for the products that the scrape collected. A URL that is not a product page, or that does not exist any more, gives no product and costs nothing. The URL results show which URLs gave a product. When the job is COMPLETED, the data is in the export files.

4. Check the result of each URLCopy Anchor Link

When the scrape settles, the job gets url_results. This field counts the URLs of the job for each result. A result without URLs is not in the field. The job list also shows url_results.

{
  "job_id": "jobreq_01k7e8h9j0k1m2n3p4q5r6s7t8",
  "job_type": "URL_SCRAPE",
  "status": "COMPLETED",
  "variant": "CATALOG_URLS",
  "url_count": 10000,
  "url_results": { "OK": 9620, "NOT_FOUND": 250, "PRICE_NOT_FOUND": 80, "INVALID_DATA": 50 }
}

Only the OK and the PRICE_NOT_FOUND URLs gave a product. The other URLs cost nothing. If many URLs get IGNORED_REQUEST, INVALID_DATA or UNKNOWN, write to support@deepscout.ai with the job ID.

Download the fileCopy Anchor Link

The file shows the result of each URL. Send GET /jobs/scraping/{job_id}/url-results. The API returns a temporary download URL, as for the sample data.

URL=$(curl -s "https://api.deepscout.ai/v1/jobs/scraping/$JOB_ID/url-results" \
  -H "Authorization: Bearer $SCOUT_API_KEY" | jq -r .url)
curl "$URL" -o url_results.csv

{ "url": "https://...amazonaws.com/url_results_jobreq_01k7e8h9j0k1m2n3p4q5r6s7t8.csv?X-Amz-Signature=..." }

The URL is valid for 1 hour. Do not send the Authorization header to this URL. It is a presigned storage URL that needs no key.

The file is in CSV format, in UTF-8, with a header row. It has one row for each URL of the job.

url,result,product_url
https://www.example-shop.cz/p/bio-kokosovy-olej-500-ml,OK,https://www.example-shop.cz/p/bio-kokosovy-olej-500-ml
https://www.example-shop.cz/p/vitamin-c-500,OK,https://www.example-shop.cz/p/vitamin-c-500-tablety
https://www.example-shop.cz/p/vitamin-c-1000,NOT_FOUND,
https://www.example-shop.cz/p/darkovy-poukaz,PRICE_NOT_FOUND,https://www.example-shop.cz/p/darkovy-poukaz

When the results are availableCopy Anchor Link

• The job gets the URL results when it changes to EXPORTING. The COMPLETED job keeps them.
• If no URL gave a product, the job fails with SCRAPE_FAILED and costs nothing. This job also has URL results, so the file shows the cause.
• Before the settlement, and for a job that failed before the settlement, the API answers 404 URL_RESULTS_NOT_FOUND.
• A full scrape has no URL results. For a full scrape, the API answers 404 JOB_NOT_FOUND.