# Scrape the full catalog (/en/api/full-scrape)

A full scrape collects all products of a website. Its results go to the export files of your project. A full scrape also makes a fresh **catalog**: for 31 days, [URL scrapes](/en/api/url-scrape.md) of the website update the prices in this catalog.

## 1. Read the estimate [#1-read-the-estimate]

When the analysis of the setup job is complete (`ESTIMATE_READY`), the website has a `full_scrape_estimate`. The API calculates it at each request, so it always shows the current price. Approve a full scrape after the setup is `COMPLETED` and you checked the [sample data](/en/api/sample-data.md). Read it with [`GET /websites/{website_id}`](/en/api/reference/websites/get-website.md):

```bash tab="cURL" tab-group="lang"
curl "https://api.deepscout.ai/v1/websites/example-shop.cz?project_id=$PROJECT_ID" \
  -H "Authorization: Bearer $SCOUT_API_KEY"
```

```python tab="Python"
website = api.get(f"{BASE}/websites/{website_id}", params={"project_id": project_id}).json()
print(website["full_scrape_estimate"])
```

```json title="full_scrape_estimate"
{
  "currency": "EUR",
  "unit_price_per_1000": "0.70",
  "quantity": 18400,
  "total": "12.88",
  "cap": "19.32",
  "min_charge": "0.10",
  "difficulty": "MEDIUM"
}
```

- `quantity` is the latest product count of the website. DeepScout counts the products at each full scrape. Before the first full scrape, the count is the estimate of the analysis.
- `difficulty` sets the price for 1,000 products. It depends on the protections of the website. See [Pricing](/en/api/pricing.md).
- `cap` is the maximum charge: 1.5 times the `total`.

## 2. Create the scrape job [#2-create-the-scrape-job]

Send [`POST /jobs/scraping`](/en/api/reference/scraping-jobs/create-scraping-job.md) with the website and the project, and without `urls`. The API creates a `FULL_SCRAPE` job with the same quote as `full_scrape_estimate`. The job waits in the `AWAITING_APPROVAL` status. Nothing is charged yet.

```bash tab="cURL" tab-group="lang"
curl https://api.deepscout.ai/v1/jobs/scraping \
  -H "Authorization: Bearer $SCOUT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz", "website_id": "example-shop.cz"}'
```

```python tab="Python"
job = api.post(f"{BASE}/jobs/scraping", json={
    "project_id": project_id,
    "website_id": website_id,
}).json()
print(job["job_id"], job["estimate"]["total"], job["estimate"]["cap"])
```

```json title="Response (201 Created)"
{
  "job_id": "jobreq_01k7d0f1g2h3j4k5m6n7p8q9r0",
  "job_type": "FULL_SCRAPE",
  "status": "AWAITING_APPROVAL",
  "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
  "website_id": "example-shop.cz",
  "variant": "FULL",
  "estimate": {
    "currency": "EUR",
    "unit_price_per_1000": "0.70",
    "quantity": 18400,
    "total": "12.88",
    "cap": "19.32",
    "min_charge": "0.10",
    "difficulty": "MEDIUM"
  },
  "rejected": { "invalid": 0, "other_website": 0, "duplicate": 0 },
  "rejected_urls": [],
  "created_at": "2026-10-13T12:00:00.000Z",
  "updated_at": "2026-10-13T12:00:00.000Z"
}
```

A website has only one unfinished full scrape in a project. If one exists, the API returns it with `200 OK` and does not create a new job.

A quote that you do not approve for 7 days changes to `CANCELLED`. To remove a quote before that, send [`POST /jobs/{job_id}/cancel`](/en/api/reference/jobs/cancel-job.md).

## 3. Approve the quote [#3-approve-the-quote]

Send [`POST /jobs/scraping/{job_id}/approve`](/en/api/reference/scraping-jobs/approve-scraping-job.md). The approval holds the `cap` from your credit and starts the scrape.

```bash tab="cURL" tab-group="lang"
curl -X POST https://api.deepscout.ai/v1/jobs/scraping/jobreq_01k7d0f1g2h3j4k5m6n7p8q9r0/approve \
  -H "Authorization: Bearer $SCOUT_API_KEY"
```

```python tab="Python"
response = api.post(f"{BASE}/jobs/scraping/{job['job_id']}/approve")
if response.ok:
    job = response.json()  # status RUNNING
else:
    error = response.json()
    print(response.status_code, error["code"], error["detail"])
```

The approval calculates the price again with the newest data of the website. These answers need an action from you:

| Status | Code | What to do |
| --- | --- | --- |
| `409` | `ESTIMATE_CHANGED` | The price went up. The response and the job have the new `estimate`. If you accept it, approve again. |
| `402` | `INSUFFICIENT_FUNDS` | Your spendable credit is less than the `cap`. Top up at least `missing_topup` (and at least 5 EUR), then approve again. See [Credits and billing](/en/api/credits.md). |
| `403` | `MONTHLY_LIMIT_REACHED` | The job does not fit in the monthly spend limit of your customer. A top-up does not help. |
| `409` | `CATALOG_DELIVERED` | Your project already got the newest full scrape of the website, and that scrape ended less than 18 hours ago. |
| `409` | `JOB_NOT_AWAITING_APPROVAL` | The job is not a quote any more, or another full or `URLS_ONLY` scrape of the website runs in your project. Wait until it ends. |

If the new price is lower, the approval accepts it and you pay the lower price.

```json title="409 ESTIMATE_CHANGED"
{
  "code": "ESTIMATE_CHANGED",
  "detail": "the price of the job went up; approve its new quote",
  "estimate": {
    "currency": "EUR",
    "unit_price_per_1000": "0.70",
    "quantity": 21000,
    "total": "14.70",
    "cap": "22.05",
    "min_charge": "0.10",
    "difficulty": "MEDIUM"
  }
}
```

## 4. Follow the progress [#4-follow-the-progress]

Read the job with [`GET /jobs/scraping/{job_id}`](/en/api/reference/scraping-jobs/get-scraping-job.md). The job goes through these statuses:

| Status | What happens |
| --- | --- |
| `APPROVED` | The approval is saved and the API starts the scrape. The approve response usually shows `RUNNING` already. |
| `RUNNING` | The scrape runs. `scrape_progress` shows the phase. |
| `SETTLING` | The scrape is complete. The API calculates the charge. |
| `EXPORTING` | The API writes the export files of the project. |
| `COMPLETED` | The data is in the export files. `settlement` shows the charge. |

While the status is `RUNNING`, `scrape_progress.phase` shows the step:

| `phase` | Console label | Description |
| --- | --- | --- |
| `COLLECTING_PAGES` | Downloading pages | The scraper downloads the product pages. `pages` counts them, and `percent` compares them with `pages_estimate`. |
| `PROCESSING_DATA` | Data cleaning and pre-processing | The API extracts and cleans the product data. |
| `UPDATING_CATALOG` | Updating the product catalog | The API merges the new data into the catalog of the website. |

```json title="A running full scrape"
{
  "job_id": "jobreq_01k7d0f1g2h3j4k5m6n7p8q9r0",
  "job_type": "FULL_SCRAPE",
  "status": "RUNNING",
  "variant": "FULL",
  "scrape_progress": {
    "phase": "COLLECTING_PAGES",
    "pages": 6900,
    "pages_estimate": 18400,
    "percent": 38
  },
  "approval": {
    "approved_by": "user_01k7c2r9q1w2e3r4t5y6z7x8c9",
    "approved_at": "2026-10-13T12:01:00.000Z"
  }
}
```

The time of a full scrape depends on the size of the catalog and on the protections of the website. A large catalog takes many hours. Read the job every 1 to 5 minutes. You also get an email when the job completes or fails.

If a full scrape of the same website runs now, or ended less than 18 hours ago, your job can use its data. This makes your job faster. The price rules do not change.

<Callout type="warn">
You cannot cancel a scrape after the approval.
</Callout>

## 5. Read the charge [#5-read-the-charge]

When the job is `COMPLETED`, `settlement` shows the charge:

```json title="settlement"
{
  "quantity": 17950,
  "currency": "EUR",
  "amount": "12.57",
  "capped": false,
  "settled_at": "2026-10-13T15:40:00.000Z"
}
```

- `quantity` is the number of products that the scrape actually collected.
- `amount` is `quantity` × `unit_price_per_1000` / 1,000, at least the `min_charge`.
- `capped` is `true` if the `cap` limited the charge.

The API releases the rest of the hold at the same time. If the scrape collects no products, the job is `FAILED` with the code `SCRAPE_FAILED`, and the charge is zero.

## 6. Get the data [#6-get-the-data]

When the job is `COMPLETED`, the export files have the new data. The dataset `all_products_with_details` has one file for each website. See [Download exports](/en/api/exports.md).

## Scrape the catalog again [#scrape-the-catalog-again]

To refresh the catalog, create and approve a new full scrape. Your project can already have the newest full scrape of the website. If that scrape ended less than 18 hours ago, the create request and the approval answer `409 CATALOG_DELIVERED`.

A full scrape also keeps your catalog fresh for URL scrapes. URL scrapes update the catalog only for 31 days after the start of the last full scrape. See [Scrape a URL list](/en/api/url-scrape.md).