Create scraping job

POST
/jobs/scraping

Create a scrape job with its estimate, awaiting the user's approval. An unfinished FULL scrape job of the website in the project is returned instead, with HTTP 200.

The API removes invalid, duplicate and other-website URLs from a URL list: rejected counts them and rejected_urls lists the first 100 removed ones.

Authorization

HTTPBearer
headerAuthorizationBearer <token>

An API key

Request Body

application/json
  1. body
website_id*string
project_id*string
urls?array<string>|null
urls_only?boolean
Defaultfalse

Response Body

The unfinished FULL scrape job of the website in the project, instead

application/json
  1. response

The created job, with the URLs that the API removed from its URL list. A FULL scrape and a clean list have zero counts. rejected_urls has the first invalid and other-website URLs.

job_id*string
job_type*string
Value in"FULL_SCRAPE""URL_SCRAPE"
status*JobRequestStatus

The lifecycle of a job. Each job type runs in its own lane.

  • WEBSITE_SETUP: CREATED, ANALYZING, ESTIMATE_READY, TRAINING, COMPLETED. The analysis and the training move the job on by themselves.
  • FULL_SCRAPE, URL_SCRAPE and MATCHING: AWAITING_APPROVAL (the user approves the quote), APPROVED, RUNNING, SETTLING (the charge), EXPORTING (the export files), COMPLETED.
  • FAILED ends a job of any lane; CANCELLED ends one that the user cancelled, or an idle one that expired.
Value in"CREATED""ANALYZING""ESTIMATE_READY""TRAINING""AWAITING_APPROVAL""APPROVED""RUNNING""SETTLING""EXPORTING""COMPLETED""FAILED""CANCELLED"
project_id*string
website_id?string|null
estimate?|null
approval?|null
settlement?|null
error?string|null
error_code?|null
contact_reason?|null
contact_request_id?string|null
created_at*string
Formatdate-time
updated_at*string
Formatdate-time
finished_at?|null
variant*ScrapeVariant

What a scrape job scrapes.

Value in"FULL""CATALOG_URLS""URLS_ONLY"
url_count?integer|null
urls?array<string>|null
scrape_progress?|null
url_results?|null
rejected*

The number of URLs that the API removed from the list, for each reason.

rejected_urls*array<string>
curl -X POST 'https://api.deepscout.ai/v1/jobs/scraping' \  -H 'Content-Type: application/json' \  -d '{  "website_id": "string",  "project_id": "string"}'
{  "job_id": "string",  "job_type": "FULL_SCRAPE",  "status": "CREATED",  "project_id": "string",  "website_id": "string",  "estimate": {    "currency": "string",    "unit_price_per_1000": "12.68",    "quantity": 0,    "total": "12.68",    "cap": "12.68",    "min_charge": "12.68",    "difficulty": "EASY",    "input_products": 0  },  "approval": {    "approved_by": "string",    "approved_at": "2026-09-16T12:30:00.000Z"  },  "settlement": {    "quantity": 0,    "currency": "string",    "amount": "12.68",    "capped": true,    "settled_at": "2026-09-16T12:30:00.000Z"  },  "error": "string",  "error_code": "WEBSITE_TOO_LARGE",  "contact_reason": "WEBSITE_TOO_LARGE",  "contact_request_id": "string",  "created_at": "2026-09-16T12:30:00.000Z",  "updated_at": "2026-09-16T12:30:00.000Z",  "finished_at": "2026-09-16T12:30:00.000Z",  "variant": "FULL",  "url_count": 0,  "urls": [    "string"  ],  "scrape_progress": {    "pages": 0,    "pages_limit": 0,    "pages_estimate": 0,    "phase": "COLLECTING_PAGES",    "percent": 0  },  "url_results": {    "property1": 0,    "property2": 0  },  "rejected": {    "invalid": 0,    "other_website": 0,    "duplicate": 0  },  "rejected_urls": [    "string"  ]}