Scrape the full catalog

A full scrape collects all products of a website. Its results go to the export files of your project. A full scrape also makes a fresh catalog: for 31 days, URL scrapes of the website update the prices in this catalog.

1. Read the estimateCopy Anchor Link

When the analysis of the setup job is complete (ESTIMATE_READY), the website has a full_scrape_estimate. The API calculates it at each request, so it always shows the current price. Approve a full scrape after the setup is COMPLETED and you checked the sample data. Read it with GET /websites/{website_id}:

curl "https://api.deepscout.ai/v1/websites/example-shop.cz?project_id=$PROJECT_ID" \
  -H "Authorization: Bearer $SCOUT_API_KEY"

{
  "currency": "EUR",
  "unit_price_per_1000": "0.70",
  "quantity": 18400,
  "total": "12.88",
  "cap": "19.32",
  "min_charge": "0.10",
  "difficulty": "MEDIUM"
}

• quantity is the latest product count of the website. DeepScout counts the products at each full scrape. Before the first full scrape, the count is the estimate of the analysis.
• difficulty sets the price for 1,000 products. It depends on the protections of the website. See Pricing.
• cap is the maximum charge: 1.5 times the total.

2. Create the scrape jobCopy Anchor Link

Send POST /jobs/scraping with the website and the project, and without urls. The API creates a FULL_SCRAPE job with the same quote as full_scrape_estimate. The job waits in the AWAITING_APPROVAL status. Nothing is charged yet.

curl https://api.deepscout.ai/v1/jobs/scraping \
  -H "Authorization: Bearer $SCOUT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz", "website_id": "example-shop.cz"}'

{
  "job_id": "jobreq_01k7d0f1g2h3j4k5m6n7p8q9r0",
  "job_type": "FULL_SCRAPE",
  "status": "AWAITING_APPROVAL",
  "project_id": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
  "website_id": "example-shop.cz",
  "variant": "FULL",
  "estimate": {
    "currency": "EUR",
    "unit_price_per_1000": "0.70",
    "quantity": 18400,
    "total": "12.88",
    "cap": "19.32",
    "min_charge": "0.10",
    "difficulty": "MEDIUM"
  },
  "rejected": { "invalid": 0, "other_website": 0, "duplicate": 0 },
  "rejected_urls": [],
  "created_at": "2026-10-13T12:00:00.000Z",
  "updated_at": "2026-10-13T12:00:00.000Z"
}

A website has only one unfinished full scrape in a project. If one exists, the API returns it with 200 OK and does not create a new job.

A quote that you do not approve for 7 days changes to CANCELLED. To remove a quote before that, send POST /jobs/{job_id}/cancel.

3. Approve the quoteCopy Anchor Link

Send POST /jobs/scraping/{job_id}/approve. The approval holds the cap from your credit and starts the scrape.

curl -X POST https://api.deepscout.ai/v1/jobs/scraping/jobreq_01k7d0f1g2h3j4k5m6n7p8q9r0/approve \
  -H "Authorization: Bearer $SCOUT_API_KEY"

The approval calculates the price again with the newest data of the website. These answers need an action from you:

If the new price is lower, the approval accepts it and you pay the lower price.

{
  "code": "ESTIMATE_CHANGED",
  "detail": "the price of the job went up; approve its new quote",
  "estimate": {
    "currency": "EUR",
    "unit_price_per_1000": "0.70",
    "quantity": 21000,
    "total": "14.70",
    "cap": "22.05",
    "min_charge": "0.10",
    "difficulty": "MEDIUM"
  }
}

4. Follow the progressCopy Anchor Link

Read the job with GET /jobs/scraping/{job_id}. The job goes through these statuses:

While the status is RUNNING, scrape_progress.phase shows the step:

{
  "job_id": "jobreq_01k7d0f1g2h3j4k5m6n7p8q9r0",
  "job_type": "FULL_SCRAPE",
  "status": "RUNNING",
  "variant": "FULL",
  "scrape_progress": {
    "phase": "COLLECTING_PAGES",
    "pages": 6900,
    "pages_estimate": 18400,
    "percent": 38
  },
  "approval": {
    "approved_by": "user_01k7c2r9q1w2e3r4t5y6z7x8c9",
    "approved_at": "2026-10-13T12:01:00.000Z"
  }
}

The time of a full scrape depends on the size of the catalog and on the protections of the website. A large catalog takes many hours. Read the job every 1 to 5 minutes. You also get an email when the job completes or fails.

If a full scrape of the same website runs now, or ended less than 18 hours ago, your job can use its data. This makes your job faster. The price rules do not change.

5. Read the chargeCopy Anchor Link

When the job is COMPLETED, settlement shows the charge:

{
  "quantity": 17950,
  "currency": "EUR",
  "amount": "12.57",
  "capped": false,
  "settled_at": "2026-10-13T15:40:00.000Z"
}

• quantity is the number of products that the scrape actually collected.
• amount is quantity × unit_price_per_1000 / 1,000, at least the min_charge.
• capped is true if the cap limited the charge.

The API releases the rest of the hold at the same time. If the scrape collects no products, the job is FAILED with the code SCRAPE_FAILED, and the charge is zero.

6. Get the dataCopy Anchor Link

When the job is COMPLETED, the export files have the new data. The dataset all_products_with_details has one file for each website. See Download exports.

Scrape the catalog againCopy Anchor Link

To refresh the catalog, create and approve a new full scrape. Your project can already have the newest full scrape of the website. If that scrape ended less than 18 hours ago, the create request and the approval answer 409 CATALOG_DELIVERED.

A full scrape also keeps your catalog fresh for URL scrapes. URL scrapes update the catalog only for 31 days after the start of the last full scrape. See Scrape a URL list.