Core concepts
Customers, projects, websites, jobs and credit, and how they connect.
Customer
A customer is the account that pays. It holds the prepaid credit, the invoices and the quota of free analyses. Your first sign-in creates a personal customer, and you are its OWNER. A customer can have more users, each with a role.
Get the customers that you belong to with GET /customers.
Project
A project keeps websites, jobs and exports separate from other projects. For example, use one project for each market or for each of your clients. Each project belongs to one customer, and the jobs of the project use the credit of this customer.
Every account has a default project. You can create more projects with POST /customers/{customer_id}/projects, up to 50 for each customer. Most requests need a project_id.
Each project has a localization: a country and a currency. The exports show the prices of all websites in the currency of the project. See Localization.
Website
A website is one online store for one country and one currency. For example, the Czech and the Slovak versions of the same store are two websites. The setup job identifies the store by its domain, so a path or a query in the URL has no effect. The analysis finds the home page of the store for the country and the currency. This home page can be on another domain than the URL that you send.
The website ID is a string such as example-shop.cz. Use it as an opaque value. Do not build it from the URL yourself.
A website is in your project after its setup job. GET /websites lists the websites of a project. Each website shows the job that sets its current state, and the estimate for a full scrape.
Job
All work runs as jobs. A job runs in the background, and you read its status to follow it. There are four types:
| Job type | What it does | Cost |
|---|---|---|
WEBSITE_SETUP | Analyzes a website, trains the scraper and makes sample data. | Paid, 10 free each month |
FULL_SCRAPE | Scrapes the full product catalog of a website. | Paid |
URL_SCRAPE | Scrapes a list of product URLs. | Paid |
MATCHING | Finds the same products on different websites. | Paid, one free sample for each project |
Paid jobs do not start immediately. The API first gives a quote, and the job waits in the AWAITING_APPROVAL status. When you approve the quote, the job starts. See Job lifecycle.
Estimate and charge
The quote of a paid job is its estimate:
quantity: the number of products in the estimate.unit_price_per_1000: the price for 1,000 products.total: the estimated price of the job.cap: the maximum charge, 1.5 times thetotal.
When you approve the job, the API holds the cap from your credit. When the job ends, the API charges the products that the job actually processed, never more than the cap. Then it releases the rest of the hold. The charge is in the settlement of the job. See Pricing.
Credit
Paid jobs use prepaid credit in EUR. You buy credit with a top-up by card, and you get an invoice for each top-up. The balance has two parts: the credit that running jobs hold, and the credit that you can spend. See Credits and billing.
Exports
Each project has export files. When a scrape or a matching completes, the API writes the files again with the new data. You download them as gzipped JSON Lines. See Download exports.
How they connect
Customer ─── credit, invoices, analyses quota
└── Project ─── exports
├── Website ─── sample data
│ ├── setup job
│ ├── full scrape jobs
│ └── URL scrape jobs
└── matching jobs (between websites of the project)