Download exports

The export files contain all the data of a project. When a scrape or a matching completes, the API writes the files again with the new data. The job stays in the EXPORTING status until the files are ready, and then changes to COMPLETED.

DatasetsCopy Anchor Link

A self-service project has these datasets:

The datasets all_products_with_products and prices_and_availability_by_website are only for managed customers. To compare your prices with your competitors, join matches with the product files of the websites. Use website_id and product_id on both sides of a pair.

The files are JSON Lines: one JSON object for each row, one row on each line. They are compressed with gzip (.jsonl.gz).

List the filesCopy Anchor Link

Send GET /exports with the project_id:

curl "https://api.deepscout.ai/v1/exports?project_id=$PROJECT_ID" \
  -H "Authorization: Bearer $SCOUT_API_KEY"

{
  "enabled": true,
  "generating": false,
  "exports": [
    {
      "dataset": "all_products_with_details",
      "website_id": "example-shop.cz",
      "file_name": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz_all_products_with_details_example-shop.cz.jsonl.gz",
      "download_url": "https://api.deepscout.ai/v1/exports/all_products_with_details?project_id=proj_01k7c2r9v8e3s0b1x6m4q5n2tz&website_id=example-shop.cz",
      "exists": true,
      "size_bytes": 11500000,
      "updated_at": "2026-10-13T15:45:00.000Z"
    },
    {
      "dataset": "matches",
      "file_name": "proj_01k7c2r9v8e3s0b1x6m4q5n2tz_matches.jsonl.gz",
      "download_url": "https://api.deepscout.ai/v1/exports/matches?project_id=proj_01k7c2r9v8e3s0b1x6m4q5n2tz",
      "exists": false
    }
  ]
}

Download a fileCopy Anchor Link

Send GET /exports/{dataset}, or use the download_url from the list. The API answers with a redirect (302) to a temporary storage URL. Let your HTTP client follow the redirect. The all_products_with_details files also need website_id.

curl -L "https://api.deepscout.ai/v1/exports/all_products_with_details?project_id=$PROJECT_ID&website_id=$WEBSITE_ID" \
  -H "Authorization: Bearer $SCOUT_API_KEY" \
  -o products.jsonl.gz

The temporary URL is valid for 1 hour. If you only need the URL, for example to give it to another system, send GET /exports/{dataset}/url. It returns {"url": "..."} instead of a redirect.

Read a fileCopy Anchor Link

Each line of the file is one JSON object.

gunzip -c products.jsonl.gz | head -n 1 | jq .

The rows are not in random order. Thus the first rows of a file are not a sample of the data.

The Datasets pages describe the fields of each dataset.

Know when new files are readyCopy Anchor Link

• A scrape or a matching job is COMPLETED only after the export files have its data. Wait for COMPLETED, then download.
• generating in GET /exports is true while an export runs. When it changes to false, the files are new.
• updated_at of each file shows when the API wrote it.

The API writes the files only after a job. There is no scheduled export.

ErrorsCopy Anchor Link