Skip to main content

Website Sources

Not available yet

Website sources are switched off in this release. GET /v1/knowledge_module → web.enabled is false, the portal's Websites tab says "Adding and syncing websites is turned off on this platform." and offers no Add website, and adding, previewing or syncing a website is refused with 403 web_sources_disabled. Upload your documents as files instead. This page describes how website sources work once your platform switches them on.

Besides uploading files, you can fill a knowledge base from a website, typically your own product documentation or help center. You give the platform a start URL; its crawler, InfusBot, discovers the pages in the scope you choose, fetches them politely and turns every page into an ordinary document of the knowledge base. Search, grounded chat, citations, metadata, deletion and billing then work exactly as they do for uploaded files.

A website source is added in the portal (open a knowledge base and choose Add website) or with the API (/v1/knowledge_bases/{kb_id}/sources, tag Web sources in the Knowledge API Reference). Adding, syncing, pausing and deleting sources needs knowledge:write (portal: admins and owners); every member can see sources, their runs and their pages.

Availability

GET /v1/knowledge_module → web tells you whether website sources are available to your organization (enabled), your effective limits (limits), the current attestation text (attestation), the terms version (terms) and InfusBot's identity (bot). When your platform operator switches website sources off, adding, previewing and syncing are refused with 403 web_sources_disabled; pages already indexed stay searchable, and you can still read, pause and delete sources and resolve held runs.

How it works​

  1. Preview. The platform looks at the site without adding it: is it reachable over https, what does robots.txt say, is there an llms.txt or a sitemap, how many pages are in scope, is there an AI-usage opt-out or bot protection.
  2. Add. You choose the scope, confirm the attestation (you are authorized to index this site) and add the source. The first sync starts at once (or, for your own site with an AI-usage opt-out, after you verify the domain).
  3. Sync. A run discovers the pages, fetches them within the site's shared request budget, and processes every new or changed page into a document. Unchanged pages are skipped and cost nothing.
  4. Keep it current. Start a new sync whenever the site changes, or verify the domain and let a scheduled refresh do it. Pages that disappeared are removed; if a run would remove more than 20 % of the source's pages (the platform default), the removals wait for your decision.

Adding a website​

In the portal​

Open the knowledge base, go to Websites and choose Add website. The wizard has two steps:

  1. Website. Enter the start address and choose Check website: the portal runs the preview. An http:// address is refused with a hint to use the https:// address. A failed check, a robots.txt block or an AI-usage opt-out stays on this step with one message and the way out.
  2. Review. A completed check opens Review and add by itself. It lists every setting that will apply: the address, which pages (Pages from), the page cap (Up to N pages (M found), with the higher cap of a verified domain), Sections built from the sample pages (leave a section out to exclude it), patterns and link depth. Advanced (collapsed) holds the name, the scope mode (This section, The whole host or Domain and subdomains), include and exclude patterns, Maximum pages and Link depth; What we found (collapsed) holds the preview details (how pages are found, robots.txt, AI-usage signals, sample pages). After a scope change, Check again gives a fresh page count. Next to the button the cost is stated: fetching is free, pages are billed like uploads (processing, then storage), and there is no estimate before the first sync. Tick the authorization statement (on your organization's first website, and after the terms change, the same box also accepts the web source terms). Start the first sync now is ticked by default. Add website adds it.

The website then appears on the Websites tab with its status, its domain tier, its last sync and its page count against the page cap. Open it to see its Sync history and Pages, and to use Sync now, Pause, Resume and Delete. Every member can see websites; only admins and owners can change them.

When the preview finds an AI-usage opt-out on a domain you have not verified, a sync would stop at once. If it is your own site, choose Add and verify first: the website is added without a sync (the button reads Add and verify the domain), the portal takes you to the domain verification screen, and once the domain is verified you start the first sync there with Start sync. When robots.txt disallows the start URL, the website cannot be added at all: verification never overrides robots.txt, so allow InfusBot in robots.txt first and check the website again.

With the API​

1. Preview the site (optional, but it tells you what a sync will find before any source or page is created):

curl -X POST "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources:preview" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"base_url": "https://docs.example.com/guide/"}'

The answer is 202 with a preview id and status: running. Poll it until status is completed or failed (usually a few seconds):

curl "https://api.infus.ai/v1/knowledge_bases/$KB_ID/source_previews/$PREVIEW_ID" \
-H "Authorization: Bearer $API_KEY"
Preview fieldMeaning
result.robotsverdict (allowed, disallowed, unavailable_disallow), whether the start URL itself is allowed, and any Crawl-delay
result.discoveryWhether llms.txt, llms-full.txt and a sitemap were found, and which discovery methods apply
result.pages_foundIn-scope URLs that robots.txt allows, without the URLs skipped for their extension (see How pages are discovered; counted up to 50,000; pages_found_estimated is true when there is no llms.txt or sitemap and the count comes from the start page's links only)
result.sample_urlsUp to 20 URLs a sync would fetch
result.robots_blocked_count, robots_blocked_sampleURLs robots.txt keeps InfusBot away from
result.ai_signalAI-usage signals found and their effect for your tier (none, warning, blocked)
result.tier, tier_page_cap, pages_to_indexYour tier, its page cap, and how many pages the first sync would index
result.bot_protectiontrue when a bot-protection challenge answered: a sync will be blocked
failure.codeFor a failed preview: https_required (no https answer on port 443), bot_protection, fetch_error or internal_error

A preview is kept for 1 hour. Previews are limited per organization per hour (429 rate_limited, limit_name: web.previews_per_hour).

2. Add the source. Read the current attestation text version and terms version from GET /v1/knowledge_module → web.attestation.text_version and web.terms.version, then:

curl -X POST "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: add-docs-site-1" \
-d '{
"type": "website",
"base_url": "https://docs.example.com/guide/",
"scope": {"mode": "path", "exclude": ["/guide/archive/**"]},
"attestation": {"accepted": true, "text_version": "web-attestation-2026-09-v1"},
"terms_acceptance": {"accepted": true, "version": "web-terms-2026-09-v1"},
"preview_id": "'"$PREVIEW_ID"'"
}'
  • attestation is required on every source. Send accepted: true only after the person adding the source has read and accepted the text; see Responsibility and attestation.
  • terms_acceptance is required on your organization's first website source, and again after the website-source terms change (web.terms.accepted is then false).
  • preview_id is optional. With it, a site whose robots.txt disallows the start URL is refused at once (422 robots_disallowed), whatever start_sync says. An AI-usage opt-out found on an unverified domain refuses only start_sync: true (422 ai_signal_optout): with "start_sync": false the source is added and nothing is fetched, so you can verify the domain first and then start the first sync. Without preview_id, the first run finds the same and the source becomes blocked_by_site.
  • start_sync defaults to true: the response is 201 with the queued first run in last_run. With false the source is added without a run.
  • A source added under a domain your organization has already verified (and whose whole scope the verification covers) starts with tier: verified.

3. Follow the run with GET …/sources/{source_id} (the last_run field), GET …/sources/{source_id}/runs, or the knowledge.source.sync_completed webhook.

Scope​

The scope decides which URLs belong to the source. It is checked again on every redirect of a page (a redirect out of scope is not followed). robots.txt and /.well-known/tdmrep.json are the exception: their redirects may lead to another path or host of the same registrable domain (for example example.com → www.example.com), still only over https to public addresses and never to a blocked domain. A robots.txt that redirects to another registrable domain is treated as unreachable, which means everything is disallowed.

FieldDefaultMeaning
modepathpath: same host, and the URL path starts with path_prefix. host: the whole host. subdomains: the start host without a leading www., and every host under it.
path_prefixthe start URL's directoryUsed by mode: path (/guide/ for https://docs.example.com/guide/intro).
include, excludenoneUp to 20 glob patterns each, matched against the normalised path and query. * matches any characters except /, ** matches any characters, ? matches one character except /. The whole path must match, an exclude wins over an include, and regular expressions are not supported.
max_pagesthe tier's page capYour own, lower cap (1 to 5,000). A value above the tier cap is stored and applies when the tier allows it; effective_max_pages on the source shows the cap in force.
max_depth5Link hops from the start URL, at most web.limits.max_depth (5 by default; a higher value is refused with 422 validation_failed). URLs from llms.txt and sitemaps count as depth 0.
excluded_urlsnoneUp to 1,000 exact URLs that are never requested directly or indexed (content reached through a redirect or a canonical URL that lands on one is discarded).

Change the scope later with PATCH …/sources/{source_id}: each top-level key you send replaces that key (lists as a whole) and the keys you leave out keep their value. A scope change applies from the next run; pages that fall out of scope are then removed, subject to the deletion hold.

Limits​

Platform defaults below; your effective values are in GET /v1/knowledge_module → web.limits. Your platform operator can lower them; they are the same for every organization (plans never change them).

LimitDefaultWhen exceeded
Pages per source, unverified domain100The run ends completed with stop_reason: page_cap_reached; extra pages are skipped (page_cap)
Pages per source, verified domain5,000Same
Sources per knowledge base5409 resource_limit_reached (limit_name: max_sources_per_kb)
Link depth (max_depth)5Deeper links are not followed
Page size (HTML, text, Markdown)10 MiB after decompressionPage skipped (size)
Linked documents (PDF, DOCX, PPTX, XLSX, CSV)50 MB, like uploadsPage skipped (size)
Fetching time per run4 hoursThe run ends completed with stop_reason: time_cap_reached; pages already fetched are processed
Previews per organization30 per hour429 rate_limited
Manual syncs of one sourceone per 60 minutes429 rate_limited with Retry-After
Domain verification checks per organization30 per clock hour (fixed, not in web.limits)429 rate_limited (limit_name: web.verifications_per_hour)
Scheduled refresh intervalat least 24 hours, at most 720422 validation_failed
URL length2,048 characters, at most 20 path segmentsPage skipped (trap)

The page cap counts the pages a source holds (indexed, processing or waiting to be removed), across runs, so a source never grows past it. The same number also caps the pages one run fetches: pages the source already holds are re-checked within that per-run fetch cap. skipped_by_reason.page_cap counts only in-scope URLs beyond the cap. When a run starts while the source holds pages and has room for more (after the cap went up, for example a higher max_pages or a verified domain, or after pages were removed), URLs the source has never seen are fetched first until the room is full or none is left; the URLs it already knows (re-checks, and earlier skips and failures) follow within the rest of the per-run fetch cap. The 1,000 documents per knowledge base limit counts uploaded documents only: website pages count against their source's page cap instead.

A source is unverified until you prove that you control its domain. Domain verification raises the cap to 5,000 pages and allows scheduled refresh.

https only​

Website sources use https on port 443 only; plain http (port 80) is never contacted.

  • A start URL with http:// is refused with 422 https_required; details.suggested_url holds the same URL with https://.
  • http:// links found on the site are fetched over https. A host that does not answer https, or a redirect to http://, skips the page (https_required).
  • Other schemes, ports other than 443, IP addresses, user names or passwords in the URL, and internal host names are refused with 422 source_url_rejected (details.rule names the rule). Every address a host name resolves to must be public, on every request and every redirect, so a site can never point InfusBot at a private network.

Domain verification​

Verifying a domain proves that your organization controls the site. It is optional while the platform accepts unverified websites: an unverified website then works, within the smaller limits. When the platform accepts verified websites only, add the website without a sync (start_sync: false; in the portal the first sync is then offered after verification), verify the domain, then sync; a sync of an unverified source is refused with 403 domain_not_verified.

Unverified domainVerified domain
Pages per source1005,000
SyncsManual onlyManual and scheduled refresh
AI-usage opt-out on the siteStops the sync (blocked_by_site, ai_signal_optout)Shown as a warning; the sync continues
Request pace on the site (shared by all customers)About 1 request per second, 2 at a timeUp to 5 requests per second, 4 at a time
JavaScript renderingNot availableAvailable per source, when the platform has it switched on
robots.txt and bot protectionAlways obeyedAlways obeyed: verification never changes this

A verification belongs to your organization and one registrable domain (for example example.com). It is never shared with another organization, and one verification covers every website source of your organization whose whole scope it covers, in any knowledge base. A source added later under a verified domain starts verified.

How to verify​

Verify the domain once, in the portal under Settings → Integrations → Domains, or with the platform Domains API. That page explains the three methods (DNS record, file, meta tag) and what makes a check fail.

  • Whole scope. The DNS record proves the registrable domain and every host below it. The file and the meta tag prove one host, so they verify a source with scope.mode path or host on exactly that host. A source with scope.mode: subdomains needs the DNS record.
  • InfusBot must be able to fetch the file or page. The file and tag checks follow the site's robots.txt (the InfusBot group, else *) and do not get around bot protection. The DNS method sends no request to the site.

The source's verification shows the result:

verification.stateMeaning
unverifiedNo verified domain of your organization covers the source's whole scope.
verifiedA verified domain covers it: domain_id, domain, method and verified_at of that domain. The state of each check (pending, failed, lost) is on the Domains page and API.

A successful check changes the tier only. It never starts a sync and never restarts a stopped source: the portal then offers Start sync (or Resume, for a source an AI-usage opt-out had stopped). With the API, start a sync, or set a blocked_by_site source active first.

Keeping the verification​

Leave the record, file or tag in place. The platform checks it again at least once a day and before every scheduled refresh (unless a check succeeded within the last hour). A proof that is gone loses the verification at that check, unless another proof of the domain still holds; a check that cannot decide (the site or DNS does not answer) keeps it for up to 72 hours. See Keeping a domain verified.

When a verification is lost, every source it alone covered drops to the unverified tier at once:

  • the page cap is 100 again: nothing is deleted, but a source above 100 pages gains no new page until it is below the cap;
  • a switched-on schedule is paused with schedule.paused_reason: verification_lost;
  • AI-usage opt-outs stop syncs again;
  • a run already under way finishes with the tier it started with.

The organization owner gets one Domain verification lost email per domain and day; no webhook is sent. Verifying the domain again restores the verified tier, and a schedule paused verification_lost resumes by itself.

Scheduled refresh​

A source on a verified domain can refresh itself on a schedule. Each scheduled run is an ordinary run (trigger: scheduled) that fetches only what may have changed, so unchanged pages cost nothing.

Turn it on on the website page in the portal (Scheduled refresh → Refresh this website automatically, then How often: every day up to every 30 days), or with the API:

curl -X PATCH "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources/$SOURCE_ID" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"schedule": {"enabled": true, "interval_hours": 24}}'
  • interval_hours is at least web.limits.min_refresh_hours (24 by default) and at most 720. Without it, the minimum is stored (daily).
  • schedule.next_run_at is the next refresh: one interval after you switch the schedule on or change the interval, plus a small fixed offset per source (under an hour), so schedules switched on together do not all start at once. The run can start a few minutes after that time: at most one scheduled run starts per site at a time, and none while the site already has two runs of any customer under way.
  • Only a verified source can have a schedule: switching it on or changing it for an unverified source answers 403 domain_not_verified. Switching it off ({"schedule": {"enabled": false}}, or Turn off) always works; next_run_at is then null.
  • A scheduled run never overlaps another run of the source. When a run cannot start (the source is paused, held, blocked or syncing, the knowledge base is not active, billing refuses the work, website sources are switched off), that refresh is skipped without an error and next_run_at moves on by whole intervals to the next time in the future (keeping the schedule's rhythm).

What a scheduled run fetches. Before starting, the run confirms the domain verification (see Keeping the verification). An indexed page whose sitemap lastmod has not changed since its content was last confirmed, whose links are known from its last full read, and that was fetched within the last 30 days, is not requested at all (it counts as not_modified). Other pages are normally requested with If-None-Match / If-Modified-Since (pages that are rendered are requested without them), and a page whose text did not change gets no new version and costs nothing. A manual Sync now never skips a page for its sitemap lastmod, so use it to force a check (within the run's page and time caps).

When a schedule pauses​

schedule.paused_reason tells you why no scheduled run starts:

paused_reasonWhyHow to turn it back on
idleNobody searched the knowledge base or used it in chat for 30 days (the platform default), counted from the last query or from when you last switched the schedule on. The source itself is unchanged: its pages stay searchable and Sync now still works. No email is sent.Turn back on in the portal, or PATCH with {"schedule": {"enabled": true}}. This restarts the 30 days.
verification_lostThe domain verification was lost.Verify the domain again: the schedule then resumes by itself. Or turn the schedule off.

Treat a paused_reason you do not know as "paused".

How pages are discovered​

Each run looks for pages in this order, and fetches them in the same order of priority (after the never-seen URLs, when the source has room for new pages; see Limits):

  1. llms.txt at the root of the site: a list of LLM-ready pages that many documentation platforms publish. Its links come first; Markdown versions of pages are preferred when offered. An llms-full.txt is detected and reported, but not indexed, because it repeats the pages.
  2. Sitemaps: the Sitemap: lines of robots.txt, then /sitemap.xml (sitemap indexes included). Newer pages (by lastmod) come first.
  3. Links: starting from the start URL, links are followed within the scope, up to the link depth.

When a site has more pages than the cap, the cut follows this order: llms.txt pages first, then sitemap pages, then linked pages. The run reports it (page_cap_reached); nothing is dropped silently. Publishing an llms.txt or a sitemap is the best way to control which pages are indexed first.

Along the way:

  • Query strings and fragments are dropped, so tracking and session parameters never create duplicate pages. A <link rel="canonical"> on the same host is honoured.
  • Links to PDF, DOCX, PPTX, XLSX, CSV, Markdown and text files inside the scope are fetched and processed like uploads of that type. Other content types are skipped (content_type).
  • Calendar-like URL patterns, endlessly repeated path segments and similar crawler traps are skipped (trap). Pages with the same text as another page of the source are kept once (duplicate).
  • URLs whose path ends in an extension the platform does not index (images, stylesheets, scripts, JSON and XML feeds, fonts, archives, audio, video, other binaries, and document formats other than the ones above, such as .doc, .xls, .ppt, .odt, .rtf or .epub) are skipped as content_type without being requested; they get no page entry and are not counted in a preview's pages_found. Any other extension is requested, and the answer's content type decides.
  • When a Markdown version listed in llms.txt asks not to be indexed (noindex), it is skipped and the page's HTML version is fetched in its place (the same address without .md; …/index.md becomes …/), still subject to the scope, robots.txt and the caps. For the rest of the run, other Markdown versions from llms.txt on that host are replaced by their HTML versions before any request.
  • Only public pages are fetched: InfusBot sends no stored cookies and no credentials, never logs in and never submits forms. By default it does not run JavaScript, so pages that only show their content after JavaScript runs are indexed with whatever text the server sent; for those, see JavaScript rendering.

JavaScript rendering​

Some sites build their content in the browser (single-page apps, client-rendered documentation): the HTML the server sends is an empty shell that scripts fill in. For such a site on a verified domain, InfusBot can render each page in an isolated browser before indexing it.

renderPortalWhat happens
off (default)OffPages are indexed from the HTML the site sends.
autoAutoOnly pages whose HTML is an empty shell (almost no text and at least one script) are rendered; the others are indexed as sent.
alwaysAlwaysEvery HTML page is rendered before indexing. Use it when the content arrives by script even though the page has other text. Slower.
  • Set it on the website page in the portal (JavaScript rendering → Render pages with JavaScript), or with render on POST …/sources or PATCH …/sources/{source_id}. The change applies from the next run; a run that is already under way keeps the mode it started with, except that switching rendering off on the platform stops new renders at once, also within a running sync.
  • When it takes effect. Only while the source's domain is verified and JavaScript rendering is switched on for the platform. It is off by default: web.render_enabled in GET /v1/knowledge_module says whether it is on. On an unverified source, or while the platform has it off, the value is kept but has no effect. render_active on the source says whether its pages are rendered now.
  • What is rendered. HTML pages only. robots.txt, sitemaps, llms.txt, linked documents (PDF and others), previews and verification checks are never rendered.
  • What the browser loads. The page itself is fetched once, under every rule above (robots.txt, the shared request budget, the signed request). While it renders, the browser may load, over https on port 443 only, scripts, stylesheets and data requests (XHR, fetch) from the page's exact origin (scheme, host and port; only where that origin's robots.txt allows InfusBot) and scripts and stylesheets from other origins. It loads no images, media, fonts or frames, opens no popup or WebSocket, and does not navigate away: a script that navigates to another page ends the render and the HTML the site sent is kept (history changes within a single-page app are fine). These requests carry InfusBot's User-Agent, count against the site's shared request budget and are not signed (see Signed requests). A render has fixed limits on requests, bytes and time; the page is taken as it is when a limit is reached.
  • Directives still apply. noindex, nofollow, noai and a TDM reservation count whether they are in the HTML the site sent or in the rendered page (the stricter one wins).
  • When a render fails (or rendering is not possible right now), the page is indexed from the HTML the site sent: a render never fails a page.
  • Each run reports render: the mode in effect for the run and how many pages were rendered, failed or unavailable (not counted: pages auto left unrendered). In the portal the sync history shows how many pages were rendered.
  • No new options. off, auto and always are the only choices: no browser setting, script, header or wait time can be passed.
  • Bot protection is still never solved: a page answered by a challenge is not rendered, and the source becomes blocked_by_site.

InfusBot and robots.txt​

InfusBot identifies itself on every request. During a sync the User-Agent is:

Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-<source-id>; +<bot page URL>)

<source-id> is your source's id. Previews and domain verification checks send the same string without the InfusBot-<source-id> part. The source's robots_token field (InfusBot-<source-id>) is what a site owner writes in robots.txt to address only your source; the portal shows it on the source page so you can pass it to the site owner.

InfusBot's public page (/bot on the platform's website; the address is the last part of the User-Agent and web.bot.info_url in GET /v1/knowledge_module) explains to site owners what InfusBot is, how to control it and how to ask for removal. Point a site owner there if they ask about your source.

Rules InfusBot always follows​

  • robots.txt is always obeyed (RFC 9309). You cannot turn this off, for your own site either: change the site's robots.txt instead.
  • The group for InfusBot-<source-id> applies if the file has one; otherwise the InfusBot group; otherwise the * group. The longest matching rule wins, and Allow wins a tie.
  • A robots.txt that answers 4xx (for example a missing file) means no restrictions. One that answers 5xx, times out or cannot be reached means everything is disallowed: if that happens for the start URL, the source becomes blocked_by_site (robots_disallowed) until you set it active again and sync.
  • robots.txt answers are cached for at most 24 hours (at most 10 minutes after a failure), so a change on the site applies to later runs within a day.
  • Crawl-delay is honoured up to 30 seconds. A delay that makes the page cap unreachable within the run's time limit ends the run with time_cap_reached.
  • noindex (in <meta name="robots"> or the X-Robots-Tag header) keeps a page out of the index; nofollow stops its links from being followed.
  • Polite pace. Every site (registrable domain, for example example.com) has one request budget shared by all customers of the platform: about 1 request per second and at most 2 requests at a time on an unverified domain (up to 5 per second and 4 at a time for sources on a verified domain; the site never sees more than the highest of these). Ten organizations adding the same site do not make ten times the load; they share the budget, so a popular site may sync more slowly. 429 and 503 answers with Retry-After are respected (up to one hour), and repeated errors pause the site for a while.

How a site owner controls InfusBot​

A site owner uses ordinary robots.txt groups. These are three alternatives, not one file: groups for the same user agent are combined, so pick the one that matches the policy you want.

Block InfusBot for every customer:

User-agent: InfusBot
Disallow: /

Allow only the documentation:

User-agent: InfusBot
Allow: /docs
Disallow: /

Block one source only (other customers keep the InfusBot or * rules):

User-agent: InfusBot-5a4b3c2d-1e0f-4a9b-8c7d-6e5f4a3b2c1d
Disallow: /

If you manage the site yourself and your robots.txt blocks crawlers in general, add an InfusBot group (or one for your source's token) that allows the pages you want indexed. InfusBot then follows only that group, not *. If you verify the domain with the verification file, allow that path too:

User-agent: *
Disallow: /

User-agent: InfusBot
Allow: /docs/
Allow: /.well-known/infus-verification
Disallow: /

Letting InfusBot through a firewall, CDN or bot protection​

robots.txt is not the only gate. A web application firewall, a CDN's bot management or a login wall can refuse InfusBot, and InfusBot never works around them (the source becomes blocked_by_site). If the site is yours, allow InfusBot there. Its name does not depend on the platform's brand: the product token InfusBot and the verification name infus-verification are always the same. It can be recognized in three ways; combine them, because a user agent alone is easy to copy.

1. User agent. Every request carries a User-Agent that contains InfusBot/1.0:

Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-<source-id>; +https://<platform-website>/bot)

Previews and verification checks send it without the InfusBot-<source-id> part. Match on InfusBot/ rather than the whole string, so a version change does not break your rule. The address at the end is InfusBot's public page (web.bot.info_url in GET /v1/knowledge_module).

2. Published IP addresses. The addresses InfusBot sends from are published as JSON next to the bot page, at https://<platform-website>/bot/ips.json, in the format other crawlers use (the address below is a documentation example):

{ "creationTime": "2026-09-28T07:46:31Z", "prefixes": [ { "ipv4Prefix": "192.0.2.10/32" } ] }
  • Only InfusBot sends from these addresses, and it sends from no other address.
  • A new address is listed at least 30 days before InfusBot starts using it, so an allowlist that follows the file is always ahead. An address InfusBot stops using stays listed for another 7 days. The exception is an emergency (an address lost or revoked): the replacement is then published and used at once, with notice the same day, and allowlists that only know the old address need updating.
  • Re-read the file at least once a day (it is cached for at most a day) instead of copying the addresses once.
  • InfusBot's production traffic never comes from an address that is not on the list. An empty prefixes list means no production address is in use yet; until one is listed, rely on the user agent and the signatures below.

3. Signed requests (Web Bot Auth). Every request InfusBot makes itself, including robots.txt, verification checks and each redirect, is signed with an HTTP Message Signature (RFC 9421) following the IETF Web Bot Auth draft (draft-ietf-webbotauth-httpsig-protocol). Each request carries three headers:

Signature-Agent: "https://<platform-website>"
Signature-Input: sig1=("@method" "@authority" "@path" "signature-agent");created=1790610300;keyid="<key id>";alg="ed25519";expires=1790610360;nonce="<random>";tag="web-bot-auth"
Signature: sig1=:<signature>:
  • The public keys are in the key directory https://<platform-website>/.well-known/http-message-signatures-directory (media type application/http-message-signatures-directory+json; Ed25519 keys as JWKs, keyid = the key's JWK SHA-256 thumbprint). The directory response is itself signed with the same keys (tag http-message-signatures-directory).
  • A signature is valid for a short time (60 seconds by default) and covers the method, the host and the path (not the query string), so it cannot be reused for another host or path; each signature also carries a random nonce, which a verifier can use to reject a repeated signature within its validity. Keys are rotated with an overlap: a new key appears in the directory at least a day before it signs, so a verifier that caches the directory for up to a day keeps working. Only a compromised key is replaced at once.
  • The scripts, stylesheets and data a page loads while it is rendered are requested with the same User-Agent and from the same published addresses, but they are not signed. A rule that requires a valid signature must therefore not apply to these requests, or JavaScript rendering of your site loses them.
  • CDNs and bot-management products that verify Web Bot Auth can use the directory to recognize InfusBot. Whether a particular CDN lists InfusBot as a verified bot depends on that CDN's own bot directory; if yours does not, allow InfusBot with a rule as below.

Example: Cloudflare. Create a WAF custom rule with the Skip action and an expression such as:

(http.user_agent contains "InfusBot/" and ip.src in {192.0.2.10})

using the addresses from ips.json (while the list is empty, the user-agent condition alone lets anyone who copies the user agent through, so limit such a rule to the paths you want indexed). Choose which security features the rule skips, for example the remaining custom rules, rate limiting rules, managed rules and, where your plan has it, Super Bot Fight Mode. The free plan's Bot Fight Mode cannot be skipped by a rule. A JavaScript challenge or "I'm Under Attack" mode stops InfusBot wherever it applies, because InfusBot never solves challenges (also not when a page is rendered), so keep InfusBot's requests out of it. Other WAFs and CDNs have equivalent allow or bypass rules keyed on user agent and source address.

After changing the site's settings, set the source active again and start a sync.

AI-usage signals​

Some sites state that their content must not be used by AI systems. InfusBot looks for these signals:

SignalWhereCounts as an opt-out when
Content Signalsa Content-Signal: line in robots.txtai-input=no
IETF AI preferencesthe Content-Usage header or robots.txtthe category covering AI use says no
TDMRep/.well-known/tdmrep.json, or the tdm-reservation: 1 header or meta taga reservation applies to the page
noai<meta name="robots"> or X-Robots-Tagalways (noimageai for image-only documents)

ai-train=no or search=no alone are recorded but are not an opt-out: the platform does not train models on your pages or run a search engine.

On an unverified domain any opt-out stops the sync: a site-wide signal (in robots.txt or tdmrep.json) as soon as those two files are read, before llms.txt, sitemaps or any page is requested; a page-level signal as soon as it is found (that page is not indexed). The source becomes blocked_by_site with status_reason: ai_signal_optout; pages indexed earlier stay until you delete them. On a verified domain the signal is shown as a warning and the sync continues. The findings are on the source (ai_signal) and on each run (ai_signals). A source added with the preview_id of a completed preview carries that preview's findings in ai_signal right away (judged for the new source's tier), so a website added with Add and verify first shows the signal before its first sync; the first run replaces them.

When a site says no: blocked_by_site​

InfusBot never gets around bot protection. It does not retry with other headers, use proxies, rotate IP addresses or user agents, solve CAPTCHAs, replay cookies or pretend to be a browser. When a site refuses, the run stops and the source becomes blocked_by_site:

status_reasonWhat happened
bot_protectionA bot-protection challenge (for example a Cloudflare, Turnstile, hCaptcha or reCAPTCHA page) answered InfusBot.
access_deniedThe start URL answered 401 or 403 on consecutive runs.
robots_disallowedrobots.txt disallows the start URL for your source, or robots.txt answered with a server error or could not be reached.
ai_signal_optoutThe site (or a page) of an unverified domain carries an AI-usage opt-out.

There are three ways out:

  1. Let InfusBot in. If you control the site, allow InfusBot in its robots.txt and in its bot-protection or firewall settings: see Letting InfusBot through a firewall, CDN or bot protection for the user agent, the published IP addresses and the signed requests. For an AI-usage opt-out on your own site, remove the opt-out, or verify the domain: the signal then becomes a warning. Verification does not change robots.txt or bot protection: those still have to allow InfusBot.
  2. Publish an llms.txt or a sitemap that lists the pages you want indexed, so discovery does not depend on crawling pages the protection guards.
  3. Upload the content as files. Export the documentation (HTML, Markdown, PDF) and upload it to the knowledge base.

After fixing the site, set the source active again (PATCH …/sources/{source_id} with {"status": "active"}, or resume it on the source page in the portal) and start a sync. The organization owner gets an email when a source is blocked, and a knowledge.source.blocked webhook is sent.

Sync runs​

A run is one pass over the site. Start one with POST …/sources/{source_id}:sync, or with Sync now on the source page in the portal; on a verified domain a scheduled refresh starts runs too. A run's trigger is manual or scheduled. Only one run per source can be active; starting another returns 409 source_sync_in_progress, and a new manual sync of the same source is possible 60 minutes after the previous one started (429 rate_limited before that).

Run statusMeaning
queued, runningDiscovering and fetching
awaiting_budgetPage processing is waiting for a paid allowance (see Pricing); no new pages are fetched meanwhile. After 24 hours without an allowance the run fails with budget_exhausted.
completedFinished. stop_reason says whether a cap cut it short.
heldFinished, but its removals wait for your decision (see the deletion hold).
stoppedThe site refused (the source is now blocked_by_site), the domain was blocked or website sources were switched off, or the knowledge base stopped being active.
failedThe site kept failing, billing refused the work, or a platform error.
cancelledYou paused or deleted the source, or a platform operator cancelled the run.

Each run reports what it found in counts: discovered, in_scope, fetched, not_modified, unchanged, new, changed, removed, pending_removal, failed, skipped and skipped_by_reason. It also records what robots.txt said (robots), the AI-usage findings (ai_signals), how long it waited for the site's shared budget (budget_wait_seconds, host_pauses) and, when it rendered pages, the JavaScript rendering summary (render). In the portal, a source's runs are on its Sync history tab.

stop_reasonRun statusMeaning
nullcompletedEverything discovered within the caps was processed.
page_cap_reachedcompletedThe source's page cap was reached; the remaining URLs were not fetched.
time_cap_reachedcompletedThe run's fetching time ran out; pages already fetched were processed.
nothing_discoveredcompletedNo in-scope URL that robots.txt allows was found.
deletion_thresholdheldRemovals above the threshold are waiting; additions and changes were published.
bot_protection, access_denied, robots_disallowed, ai_signal_optoutstoppedThe site said no; see blocked_by_site.
domain_blocked, tenant_disabled, web_disabledstoppedThe domain is blocked on the platform, website sources were switched off for your organization, or switched off on the platform.
kb_unavailablestoppedThe knowledge base stopped being active during the run (for example suspended or frozen).
host_pausedfailedThe site kept answering 429, 503 or server errors, or timing out, and the resulting pause (from Retry-After, or after repeated errors) would end after the run's fetching time limit. Shorter pauses are waited out.
budget_exhaustedfailedPages waited 24 hours for a paid allowance.
billing_blockedfailedBilling refused a page with payment_suspended; no further pages are fetched. The refused page is not charged; pages accepted before the refusal are processed and billed normally.
internal_errorfailedPlatform failure after retries; unprocessed pages are not charged.
source_paused, source_deleted, operator_cancelledcancelledYou paused or deleted the source, or a platform operator cancelled the run.

Treat a stop_reason or status_reason you do not know as "other": new values may be added.

Every page is a document​

Each page becomes one document of the knowledge base, with origin: web:

  • The document name is the page title (or its URL path); source holds its provenance: {source_id, url, fetched_at, verified}. List a source's pages as documents with GET …/documents?source_id=<source_id> (or all website pages with ?origin=web).
  • Search hits and chat citations of a website page carry the same source, so you can link the answer to the page. Label verified: false as third-party content.
  • A changed page gets a new version: the old version keeps answering until the new one is ready, exactly like a file replacement. A file replacement (PUT …/file) of a website page is refused (409 invalid_document_state): new versions come from the site.
  • Common kinds of hidden content (scripts, elements hidden with the hidden attribute, aria-hidden or inline styles such as display:none) and invisible characters (zero-width, bidirectional control and tag characters) are removed before indexing; the document then carries the hidden_content_removed warning. Page text is passed to the model as untrusted reference material, with the instruction not to follow commands in it. As with uploads, this reduces but cannot fully eliminate the risk that a page crafted to manipulate a model changes an answer (prompt injection), so index sites you trust.
  • A page is unchanged when the site answers 304 Not Modified, or when its extracted text is identical to the indexed version. An unchanged page gets no new version, sends no webhook and costs nothing.

Pages and skip reasons​

GET …/sources/{source_id}/pages lists the pages of a source with their status (queued, fetching, processing, indexed, skipped, failed, pending_removal, removed, excluded), document_id, http_status, depth, how it was found (discovered_via) and when it last changed. Filter with ?status= and ?skip_reason=.

skip_reasonMeaning
robotsrobots.txt disallows the URL.
noindexThe page asks not to be indexed.
ai_signalPage-level AI-usage opt-out on an unverified domain (the run stops too).
content_typeNot an accepted type, or the file signature disagrees. Links with an obvious non-document extension (images, stylesheets, scripts, feeds, archives and similar) get this reason without being requested.
sizeLarger than the page or file size limit, or suspiciously compressed.
scopeOutside the source's scope.
excludedMatches an exclude pattern, excluded_urls or an excluded page; also when a redirect or canonical URL lands on one.
redirect_out_of_scopeA redirect left the scope and was not followed.
https_requiredA redirect to http://, or the host does not answer https on port 443.
url_rejectedThe URL leads to an address the platform never contacts (private or internal networks).
trapThe URL looks like a crawler trap (too long, too many or repeated segments).
duplicateSame text as another page of this source.
page_capFound and in scope, but beyond the page cap.
http_errorThe site answered another 4xx or 5xx status.
fetch_errorDNS, connection or TLS failure, a timeout, a very slow answer, or too many redirects.
bot_protectionA challenge page answered this URL (the source is then blocked_by_site).

A page that answered 404 or 410 after being indexed is not a skip: it is a removal.

Removals and the deletion hold​

A page indexed by an earlier run is removed when a run finds it gone (404/410), disallowed by robots.txt, marked noindex, out of scope after a scope change, or excluded. A page that simply was not found again is removed only after a complete discovery: a run cut short by a cap, a stop or failed fetches never removes pages just because it did not reach them. Removing a page deletes its document, with the usual knowledge.document.deleted webhook.

To protect you from a broken sitemap or a site outage wiping the knowledge base, a run that would remove more than 20 % (the platform default; the run reports the threshold in force as deletion.threshold_percent) of the pages that were indexed before the run is held (exactly 20 % is not held):

  • the run ends held (stop_reason: deletion_threshold) and the source becomes held;
  • additions and changes are published normally; the pages to remove stay searchable, with page status pending_removal;
  • the organization owner gets an email, and knowledge.source.sync_completed carries the run;
  • no new run starts until you decide, and the hold never expires on its own.

Decide with:

curl -X POST "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources/$SOURCE_ID/runs/$RUN_ID:resolve" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"decision": "apply_deletions"}'
DecisionEffect
apply_deletionsThe held pages' documents are deleted (pages you deleted meanwhile are skipped).
discard_deletionsThe pages stay indexed; the next run looks at them again.

A decision is recorded once on the run (deletion.decision, deletion.decided_at); sending the same decision again returns 200, a different one returns 409 invalid_source_state. The run keeps the status held (and its counts as they were when it ended): a non-null deletion.decision is what marks it as resolved. A held source becomes active; a paused source stays paused. In the portal a held source shows Needs review and a Review page removals notice with Remove pages (apply_deletions) and Keep pages (discard_deletions); the pages stay searchable until you decide.

Excluded pages​

You can delete a website page's document like any document (DELETE …/documents/{document_id} or a batch delete). Its page then becomes excluded: no later run requests it directly or indexes it again, even if the site still links to it. If a redirect or a canonical URL leads to an excluded page, the fetched content is discarded. There is no limit on the number of excluded pages.

  • List them with GET …/sources/{source_id}/pages?status=excluded.
  • Bring pages back with PATCH …/sources/{source_id} and {"reinclude_urls": ["https://docs.example.com/guide/old-page/"]} (up to 100 URLs per call); they are eligible again from the next run. In the portal, open the website's Pages tab, filter by Excluded, select the pages and choose Re-include selected.
  • To keep URLs out before they are ever indexed, use scope.exclude patterns or scope.excluded_urls instead.

Source status​

StatusPortalMeaningWhat you can do
activeActiveIdle and readySync, pause, change, delete
syncingSyncingA run is queued, running or waiting for budgetPause (cancels the run), delete
heldNeeds reviewThe latest run's removals wait for a decision (status_reason: deletion_threshold)Resolve the run; pause; delete
blocked_by_siteBlocked by siteThe site refused (see above)Fix the cause, set active, sync
pausedPausedPaused by you (customer); or left paused when the platform lifted a block: rarely unblocked (domain block lifted) or tenant_reenabled (website sources switched back on for your organization), or takedown_lifted (the block ended after a removal request deleted the source's pages)Set active
disabled_by_adminDisabledThe platform blocked the domain (domain_blocked) or switched website sources off for your organization (tenant_disabled)Contact support; pages stay searchable, and you can still delete
deletingDeletingThe source and its pages are being deletedNothing

When neither a domain block nor the organization-wide switch-off applies any more (the platform lifted the block, or switched website sources back on for your organization), each source gets back the status it had before (active, also for a source that was syncing; paused with its reason; blocked_by_site with the site's reason; held while its removals still wait for a decision, otherwise active). No run starts, and no webhook or email is sent. Only a source whose earlier status is not known is left paused, with status_reason unblocked (domain block lifted) or tenant_reenabled (website sources switched back on): resume it when you are ready.

After a removal request that deleted pages, a source does not get its earlier status back. If the platform deleted a source's indexed pages because of a removal request (takedown) during the block, the source stays paused with status_reason: takedown_lifted when the block is lifted, even if it was active before, and has no pages. Make sure you may use the content again before you resume it with {"status": "active"}: the next sync fetches the pages again. A removal request that only blocked the domain, without deleting pages, keeps the normal behaviour above. The portal shows: "The platform deleted this website's pages after a takedown request and has since ended the block. Make sure you may use the content before you resume the website: the next sync fetches the pages again."

Pause and resume with PATCH …/sources/{source_id} and {"status": "paused"} or {"status": "active"}. Pausing cancels a running sync; indexed pages stay searchable while paused. Resuming a source whose held run is still unresolved returns it to held. Other status changes are refused with 409 invalid_source_state.

Deleting a source (DELETE …/sources/{source_id}, 202) cancels its run and deletes every page document through the normal deletion process, with one knowledge.document.deleted webhook per page. Deleting the knowledge base deletes its sources.

Pricing​

Website sources use the same meters as uploads (Pricing and Billing):

  • No crawl fee. Previews, discovery, robots.txt and sitemap reads and page fetches are free.
  • Processing tokens for each new or changed page, and storage for the page's original bytes (the fetched HTML, the rendered HTML when the page was rendered, or the linked file), exactly as for an uploaded file. Free-trial allowances apply.
  • Unchanged pages are never charged again, and neither are platform retries. JavaScript rendering adds no charge: a rendered page is billed like any other page, and not at all when its text did not change. A scheduled refresh costs the same as a manual sync: only new and changed pages are billed.
  • When processing cannot be paid (trial used up, no balance), pages wait in awaiting_budget and the run waits with them for up to 24 hours.

Responsibility and attestation​

The summary below is a plain-language overview of how responsibility is shared. It is not legal advice, and the binding text is the platform's terms of use and acceptable-use policy (the address is in web.terms.url).

  • You attest, for every source. Adding a website requires an explicit confirmation, shown next to a checkbox in the portal and required as attestation in the API. The text is versioned; web.attestation.text always has the current one. At the time of writing it reads: "I confirm that I am authorised to index the content of this website. I am responsible for this content and for how it is used, and I will indemnify the platform against third-party claims arising from it."
  • You accept the website-source terms on your organization's first source and after they change. They forbid, among other things, indexing sites you are not authorized to index, and infringing copyright or data-protection law through the indexed content.
  • What is recorded. For each source the platform records who accepted, when, from which IP address and which text version. The IP address is kept as evidence and never returned by the API.
  • The platform's own care. InfusBot identifies itself, always obeys robots.txt, never gets around bot protection, shares one polite request budget per site across all customers, stops at AI-usage opt-outs on unverified domains and fetches only public pages. Each run records what robots.txt said and what signals were found.
  • Site owners can object. A site owner can block InfusBot or a single source in robots.txt, or ask the platform to remove their content (contact details are on the bot page). On a valid request the platform blocks the domain for every customer: sources whose start URL is on that domain become disabled_by_admin with status_reason: domain_blocked, while sources with a wider scope that merely includes the domain keep running without it. If the owner asks, the indexed pages on that domain are deleted, in whichever source they are. You are notified by email and with a knowledge.source.blocked webhook. The platform can also switch website sources off for an organization that misuses them (tenant_disabled). When a block is lifted, sources get back the status they had before, except sources whose pages were deleted on the owner's request: those stay paused (takedown_lifted) until you resume them (see Source status).

Webhooks and emails​

Two webhook events cover website sources, in addition to the usual knowledge.document.* events for each page (which carry origin: web and the source_id):

EventSent when
knowledge.source.sync_completedA run ended, whatever the result: completed, held, stopped, failed or cancelled. Exactly one per run; read data.run.status and data.run.stop_reason.
knowledge.source.blockedA source became blocked_by_site or disabled_by_admin, with or without a run (for example after a removal request).

The organization owner also gets an email when a source is blocked, when a run fails, when a run is held for a deletion decision, and when a domain verification is lost. Losing a verification sends no webhook: read verification and tier on the source.

Errors​

The codes specific to website sources are listed in Error Codes: web_sources_disabled, domain_blocked, domain_not_verified, source_sync_in_progress, invalid_source_state, duplicate_source, https_required, source_url_rejected, robots_disallowed, ai_signal_optout, attestation_required and terms_acceptance_required. Verification checks are limited with 429 rate_limited (limit_name: web.verifications_per_hour), and previews and syncs also have a short per-minute burst limit per organization (limit_name: gateway.web_previews_per_minute, gateway.web_syncs_per_minute).