Skip to main content

Web sources

Index a website into a knowledge base: preview, add, sync, runs, pages and held deletions.

📄️ Discover a website without adding it (asynchronous; poll the preview)

Discovers a website without adding it: no source, page or run is stored, only the preview. Checked at once: URL rules (422 `https_required`, `source_url_rejected`), domain blocked by the platform (403 `domain_blocked`), website sources switched off (403 `web_sources_disabled`) and previews per hour (429 `rate_limited`, `limit_name: web.previews_per_hour`). The discovery itself runs asynchronously (at most 20 requests to the site, within the site's shared request budget): poll `GET …/source_previews/{preview_id}` until `status` is `completed` or `failed`, typically a few seconds. A preview is kept for 1 hour. Replaying the request with the same `Idempotency-Key` returns the original 202 body and counts once against the hourly limit.

📄️ Start a manual sync run

Starts a manual sync run. 409 `source_sync_in_progress` while a run is queued, running or waiting for budget. 409 `invalid_source_state` when the source is `paused`, `held`, `blocked_by_site`, `disabled_by_admin` or `deleting` (set it `active` first), and while an earlier held run is unresolved, even if the source was paused and resumed meanwhile. 429 `rate_limited` (`limit_name: web.manual_sync_cooldown_minutes`) when the previous run of this source started less than the cooldown ago. 403 `domain_blocked`, `web_sources_disabled` or `domain_not_verified`. The knowledge-base and billing checks of an upload apply (409 knowledge-base state codes, 402 `payment_suspended`, 503 `maintenance_mode`). Pages that did not change cost nothing.

📄️ Apply or discard the removals of a held run

Only for a run in `held` (otherwise 409 `invalid_source_state`). `apply_deletions` deletes the held pages' documents; `discard_deletions` keeps them indexed, and the next run evaluates them again. A decision is recorded once: the run keeps `status: held` and gets `deletion.decision` and `deletion.decided_at`; sending the same decision again returns 200, a different one is 409 `invalid_source_state`. Only the hold is cleared: a `held` source becomes `active`, a `paused` or `disabled_by_admin` source keeps its status, a `deleting` source refuses (409). The pages concerned are the run's `pending_removal` pages when it ended; pages you deleted meanwhile are skipped.