# Scrape a URL list (/en/api/url-scrape)

A URL scrape collects only the product pages that you send. Use it for two tasks:

- **Update a catalog.** Refresh the prices and the availability of some products often, and refresh the full catalog less often.
- **Scrape only some products.** Get the products that you need, without the full catalog.

Before a URL scrape, the website must be in your project. See [Add a website](/en/api/add-website.md).

## Variants [#variants]

The API selects the variant of each URL scrape. `variant` on the job shows it.

| `variant` | When | The result |
| --- | --- | --- |
| `CATALOG_URLS` | Your project has a fresh full catalog of the website. | The scraped products update the full catalog. The exports have the full catalog with the new data. |
| `URLS_ONLY` | Your project has no fresh full catalog, and one of these is true: it never completed a full scrape of the website, its last completed scrape of the website was `URLS_ONLY`, or you set `urls_only`. | The exports of the website have only the products of URL scrapes. |

A full catalog is **fresh** for 31 days after the start of its full scrape. It is also fresh while a full scrape of your project runs. With a fresh catalog, the variant is always `CATALOG_URLS`, and `urls_only` has no effect.

<Callout type="warn" title="URLS_ONLY replaces the catalog in your exports">
When a `URLS_ONLY` scrape completes, the catalog products of the website leave your exports. Only the products of URL scrapes stay. The next completed full scrape brings the full catalog back.
</Callout>

## 1. Create the URL scrape [#1-create-the-url-scrape]

Send [`POST /jobs/scraping`](/en/api/reference/scraping-jobs/create-scraping-job.md) with the list in `urls`. One job accepts from 1 to 100,000 URLs.

```bash tab="cURL" tab-group="lang"
curl https://api.deepscout.ai/v1/jobs/scraping \
  -H "Authorization: Bearer $SCOUT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
    "website_id": "example-shop.cz",
    "urls": [
      "https://www.example-shop.cz/p/bio-kokosovy-olej-500-ml",
      "https://www.example-shop.cz/p/vitamin-c-1000",
      "https://www.example-shop.cz/p/vitamin-c-1000",
      "https://www.other-shop.cz/p/vitamin-d3"
    ]
  }'
```

```python tab="Python"
with open("urls.txt") as file:
    urls = [line.strip() for line in file if line.strip()]

job = api.post(f"{BASE}/jobs/scraping", json={
    "project_id": project_id,
    "website_id": website_id,
    "urls": urls,
}).json()
print(job["variant"], job["url_count"], job["rejected"], job["estimate"]["total"])
```

The API cleans the list before it creates the job:

- It adds `https://` to a URL without a scheme, changes `http://` to `https://` and changes the domain to lower case.
- It removes URLs that are not valid (`invalid`).
- It removes URLs of other websites (`other_website`). The `www.` and the bare form of the domain are both accepted. Usually, other subdomains are not.
- It removes duplicates (`duplicate`) and keeps the first copy.

`rejected` counts the removed URLs for each reason. `rejected_urls` shows the first 100 invalid and other-website URLs. The job contains only the accepted URLs, and the quote is for these URLs.

```json title="Response (201 Created)"
{
  "job_id": "jobreq_01k7d5s6t7v8w9x0y1z2a3b4c5",
  "job_type": "URL_SCRAPE",
  "status": "AWAITING_APPROVAL",
  "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
  "website_id": "example-shop.cz",
  "variant": "CATALOG_URLS",
  "url_count": 2,
  "estimate": {
    "currency": "EUR",
    "unit_price_per_1000": "0.70",
    "quantity": 2,
    "total": "0.10",
    "cap": "0.15",
    "min_charge": "0.10",
    "difficulty": "MEDIUM"
  },
  "rejected": { "invalid": 0, "other_website": 1, "duplicate": 1 },
  "rejected_urls": ["https://www.other-shop.cz/p/vitamin-d3"],
  "created_at": "2026-10-13T12:00:00.000Z",
  "updated_at": "2026-10-13T12:00:00.000Z"
}
```

If the API accepts no URL, it answers `400 INVALID_URL_LIST`. An empty list, or a list of more than 100,000 URLs, gets `422`. For a larger list, split it into more jobs.

Each request creates a new job, also for the same list.

## 2. Approve the quote [#2-approve-the-quote]

Approve the job with [`POST /jobs/scraping/{job_id}/approve`](/en/api/reference/scraping-jobs/approve-scraping-job.md). The approval works as for a [full scrape](/en/api/full-scrape.md#3-approve-the-quote): it holds the `cap`, and it can answer `ESTIMATE_CHANGED`, `INSUFFICIENT_FUNDS` or `MONTHLY_LIMIT_REACHED`.

The approval also selects the variant again. If your catalog became too old since the quote, the approval answers `409 CATALOG_EXPIRED`. If a full scrape made a fresh catalog since the quote, a `URLS_ONLY` job changes to `CATALOG_URLS`.

To remove a quote, send [`POST /jobs/{job_id}/cancel`](/en/api/reference/jobs/cancel-job.md). A quote that you do not approve for 7 days changes to `CANCELLED`.

## When the catalog is too old [#when-the-catalog-is-too-old]

If the full catalog of your project is older than 31 days, the create request or the approval answers `409`. This does not occur if the last completed scrape of the website was `URLS_ONLY`.

```json title="409 CATALOG_EXPIRED"
{
  "code": "CATALOG_EXPIRED",
  "detail": "the FULL catalog of website example-shop.cz is older than 31 days"
}
```

Select one of two options:

<Tabs items={['Refresh the catalog', 'Scrape only these URLs']}>
<Tab value="Refresh the catalog">

Your URL list then updates a new full catalog.

1. Cancel the old URL quote, if you have one.
2. Create and approve a [full scrape](/en/api/full-scrape.md).
3. Create the URL scrape again. While the full scrape runs, the catalog counts as fresh, so the variant is `CATALOG_URLS`.

</Tab>
<Tab value="Scrape only these URLs">

Only the products of URL lists stay in your exports. The catalog products leave the exports until the next full scrape.

1. Cancel the old URL quote, if you have one.
2. Create the URL scrape again with `"urls_only": true`.

```json
{
  "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
  "website_id": "example-shop.cz",
  "urls": ["https://www.example-shop.cz/p/vitamin-c-1000"],
  "urls_only": true
}
```

</Tab>
</Tabs>

## 3. Follow the progress and get the data [#3-follow-the-progress-and-get-the-data]

A URL scrape has the same statuses and the same `scrape_progress` as a [full scrape](/en/api/full-scrape.md#4-follow-the-progress). While it collects pages, `percent` compares `pages` with `url_count`.

To list the URL scrapes of a website, send [`GET /jobs/scraping`](/en/api/reference/scraping-jobs/list-scraping-jobs.md) with `project_id` and `website_id`. The list shows `url_count`. The list of URLs is only in [`GET /jobs/scraping/{job_id}`](/en/api/reference/scraping-jobs/get-scraping-job.md).

URL scrapes do not change the state of the website. The `job` of the website stays the setup job or the newest full scrape.

The charge is for the products that the scrape collected. A URL that is not a product page, or that does not exist any more, gives no product and costs nothing. The [URL results](#4-check-the-result-of-each-url) show which URLs gave a product. When the job is `COMPLETED`, the data is in the [export files](/en/api/exports.md).

<Callout title="A full scrape waits for a URLS_ONLY scrape">
While a `URLS_ONLY` scrape of a website runs, you cannot approve a full scrape of the same website. Approve it when the URL scrape ends.
</Callout>

## 4. Check the result of each URL [#4-check-the-result-of-each-url]

When the scrape settles, the job gets `url_results`. This field counts the URLs of the job for each result. A result without URLs is not in the field. The [job list](/en/api/reference/scraping-jobs/list-scraping-jobs.md) also shows `url_results`.

```json title="Part of the job"
{
  "job_id": "jobreq_01k7e8h9j0k1m2n3p4q5r6s7t8",
  "job_type": "URL_SCRAPE",
  "status": "COMPLETED",
  "variant": "CATALOG_URLS",
  "url_count": 10000,
  "url_results": { "OK": 9620, "NOT_FOUND": 250, "PRICE_NOT_FOUND": 80, "INVALID_DATA": 50 }
}
```

| `result` | Description |
| --- | --- |
| `OK` | The URL gave a product with a price. |
| `PRICE_NOT_FOUND` | The URL gave a product without a price. The product counts in the charge. |
| `NOT_FOUND` | The page answered with an error status, for example 404 or 410. Usually, the product does not exist any more. |
| `IGNORED_RESPONSE` | The page is not a product page, for example a file, or it did not load after all retries. |
| `IGNORED_REQUEST` | The scraper did not download the URL. |
| `INVALID_DATA` | The page loaded, but its data gave no valid product. |
| `UNKNOWN` | The scrape has no record of the URL. |

Only the `OK` and the `PRICE_NOT_FOUND` URLs gave a product. The other URLs cost nothing. If many URLs get `IGNORED_REQUEST`, `INVALID_DATA` or `UNKNOWN`, write to [support@deepscout.ai](mailto:support@deepscout.ai) with the job ID.

### Download the file [#download-the-file]

The file shows the result of each URL. Send [`GET /jobs/scraping/{job_id}/url-results`](/en/api/reference/scraping-jobs/get-scraping-job-url-results.md). The API returns a temporary download URL, as for the [sample data](/en/api/sample-data.md).

```bash tab="cURL" tab-group="lang"
URL=$(curl -s "https://api.deepscout.ai/v1/jobs/scraping/$JOB_ID/url-results" \
  -H "Authorization: Bearer $SCOUT_API_KEY" | jq -r .url)
curl "$URL" -o url_results.csv
```

```python tab="Python"
import csv
import io

url = api.get(f"{BASE}/jobs/scraping/{job_id}/url-results").json()["url"]
text = requests.get(url).content.decode("utf-8")
for row in csv.DictReader(io.StringIO(text)):
    if row["result"] != "OK":
        print(row["result"], row["url"])
```

```json title="Response"
{ "url": "https://...amazonaws.com/url_results_jobreq_01k7e8h9j0k1m2n3p4q5r6s7t8.csv?X-Amz-Signature=..." }
```

The URL is valid for 1 hour. Do not send the `Authorization` header to this URL. It is a presigned storage URL that needs no key.

The file is in CSV format, in UTF-8, with a header row. It has one row for each URL of the job.

```csv title="url_results_jobreq_01k7e8h9j0k1m2n3p4q5r6s7t8.csv"
url,result,product_url
https://www.example-shop.cz/p/bio-kokosovy-olej-500-ml,OK,https://www.example-shop.cz/p/bio-kokosovy-olej-500-ml
https://www.example-shop.cz/p/vitamin-c-500,OK,https://www.example-shop.cz/p/vitamin-c-500-tablety
https://www.example-shop.cz/p/vitamin-c-1000,NOT_FOUND,
https://www.example-shop.cz/p/darkovy-poukaz,PRICE_NOT_FOUND,https://www.example-shop.cz/p/darkovy-poukaz
```

| Column | Description |
| --- | --- |
| `url` | A URL of the job, as the API accepted it. |
| `result` | The result of the URL. See the table above. |
| `product_url` | The URL of the product. It is different from `url` when the page redirected to another URL. It is empty when the URL gave no product. |

### When the results are available [#when-the-results-are-available]

- The job gets the URL results when it changes to `EXPORTING`. The `COMPLETED` job keeps them.
- If no URL gave a product, the job fails with `SCRAPE_FAILED` and costs nothing. This job also has URL results, so the file shows the cause.
- Before the settlement, and for a job that failed before the settlement, the API answers `404 URL_RESULTS_NOT_FOUND`.
- A full scrape has no URL results. For a full scrape, the API answers `404 JOB_NOT_FOUND`.