Scraping Jobs
Create scraping job
POST
Create a scrape job with its estimate, awaiting the user's approval. An unfinished FULL scrape job of the website in the project is returned instead, with HTTP 200.
The API removes invalid, duplicate and other-website URLs from a URL list: rejected
counts them and rejected_urls lists the first 100 removed ones.
HTTPBearerheader
AuthorizationBearer <token>An API key
application/json- body
website_id*stringproject_id*stringurls?array<string>|nullurls_only?booleanDefault
falseThe unfinished FULL scrape job of the website in the project, instead
application/json- response
The created job, with the URLs that the API removed from its URL list. A FULL scrape and
a clean list have zero counts. rejected_urls has the first invalid and other-website URLs.
job_id*stringjob_type*stringValue in
"FULL_SCRAPE""URL_SCRAPE"status*JobRequestStatusThe lifecycle of a job. Each job type runs in its own lane.
- WEBSITE_SETUP: CREATED, ANALYZING, ESTIMATE_READY, TRAINING, COMPLETED. The analysis and the training move the job on by themselves.
- FULL_SCRAPE, URL_SCRAPE and MATCHING: AWAITING_APPROVAL (the user approves the quote), APPROVED, RUNNING, SETTLING (the charge), EXPORTING (the export files), COMPLETED.
- FAILED ends a job of any lane; CANCELLED ends one that the user cancelled, or an idle one that expired.
Value in
"CREATED""ANALYZING""ESTIMATE_READY""TRAINING""AWAITING_APPROVAL""APPROVED""RUNNING""SETTLING""EXPORTING""COMPLETED""FAILED""CANCELLED"project_id*stringwebsite_id?string|nullestimate?|nullapproval?|nullsettlement?|nullerror?string|nullerror_code?|nullcontact_reason?|nullcontact_request_id?string|nullcreated_at*stringFormat
date-timeupdated_at*stringFormat
date-timefinished_at?|nullvariant*ScrapeVariantWhat a scrape job scrapes.
Value in
"FULL""CATALOG_URLS""URLS_ONLY"url_count?integer|nullurls?array<string>|nullscrape_progress?|nullurl_results?|nullrejected*The number of URLs that the API removed from the list, for each reason.
rejected_urls*array<string>curl -X POST 'https://api.deepscout.ai/v1/jobs/scraping' \ -H 'Content-Type: application/json' \ -d '{ "website_id": "string", "project_id": "string"}'{ "job_id": "string", "job_type": "FULL_SCRAPE", "status": "CREATED", "project_id": "string", "website_id": "string", "estimate": { "currency": "string", "unit_price_per_1000": "12.68", "quantity": 0, "total": "12.68", "cap": "12.68", "min_charge": "12.68", "difficulty": "EASY", "input_products": 0 }, "approval": { "approved_by": "string", "approved_at": "2026-09-16T12:30:00.000Z" }, "settlement": { "quantity": 0, "currency": "string", "amount": "12.68", "capped": true, "settled_at": "2026-09-16T12:30:00.000Z" }, "error": "string", "error_code": "WEBSITE_TOO_LARGE", "contact_reason": "WEBSITE_TOO_LARGE", "contact_request_id": "string", "created_at": "2026-09-16T12:30:00.000Z", "updated_at": "2026-09-16T12:30:00.000Z", "finished_at": "2026-09-16T12:30:00.000Z", "variant": "FULL", "url_count": 0, "urls": [ "string" ], "scrape_progress": { "pages": 0, "pages_limit": 0, "pages_estimate": 0, "phase": "COLLECTING_PAGES", "percent": 0 }, "url_results": { "property1": 0, "property2": 0 }, "rejected": { "invalid": 0, "other_website": 0, "duplicate": 0 }, "rejected_urls": [ "string" ]}