Website Sources
Website sources are switched off in this release. GET /v1/knowledge_module → web.enabled is false, the portal's Websites tab says "Adding and syncing websites is turned off on this platform." and offers no Add website, and adding, previewing or syncing a website is refused with 403 web_sources_disabled. Upload your documents as files instead. This page describes how website sources work once your platform switches them on.
Besides uploading files, you can fill a knowledge base from a website, typically your own product documentation or help center. You give the platform a start URL; its crawler, InfusBot, discovers the pages in the scope you choose, fetches them politely and turns every page into an ordinary document of the knowledge base. Search, grounded chat, citations, metadata, deletion and billing then work exactly as they do for uploaded files.
A website source is added in the portal (open a knowledge base and choose Add website) or with the API (/v1/knowledge_bases/{kb_id}/sources, tag Web sources in the Knowledge API Reference). Adding, syncing, pausing and deleting sources needs knowledge:write (portal: admins and owners); every member can see sources, their runs and their pages.
GET /v1/knowledge_module → web tells you whether website sources are available to your organization (enabled), your effective limits (limits), the current attestation text (attestation), the terms version (terms) and InfusBot's identity (bot). When your platform operator switches website sources off, adding, previewing and syncing are refused with 403 web_sources_disabled; pages already indexed stay searchable, and you can still read, pause and delete sources and resolve held runs.
How it works
- Preview. The platform looks at the site without adding it: is it reachable over https, what does robots.txt say, is there an
llms.txtor a sitemap, how many pages are in scope, is there an AI-usage opt-out or bot protection. - Add. You choose the scope, confirm the attestation (you are authorized to index this site) and add the source. The first sync starts at once (or, for your own site with an AI-usage opt-out, after you verify the domain).
- Sync. A run discovers the pages, fetches them within the site's shared request budget, and processes every new or changed page into a document. Unchanged pages are skipped and cost nothing.
- Keep it current. Start a new sync whenever the site changes, or verify the domain and let a scheduled refresh do it. Pages that disappeared are removed; if a run would remove more than 20 % of the source's pages (the platform default), the removals wait for your decision.
Adding a website
In the portal
Open the knowledge base, go to Websites and choose Add website. The wizard has two steps:
- Website. Enter the start address and choose Check website: the portal runs the preview. An
http://address is refused with a hint to use thehttps://address. A failed check, a robots.txt block or an AI-usage opt-out stays on this step with one message and the way out. - Review. A completed check opens Review and add by itself. It lists every setting that will apply: the address, which pages (Pages from), the page cap (Up to N pages (M found), with the higher cap of a verified domain), Sections built from the sample pages (leave a section out to exclude it), patterns and link depth. Advanced (collapsed) holds the name, the scope mode (This section, The whole host or Domain and subdomains), include and exclude patterns, Maximum pages and Link depth; What we found (collapsed) holds the preview details (how pages are found, robots.txt, AI-usage signals, sample pages). After a scope change, Check again gives a fresh page count. Next to the button the cost is stated: fetching is free, pages are billed like uploads (processing, then storage), and there is no estimate before the first sync. Tick the authorization statement (on your organization's first website, and after the terms change, the same box also accepts the web source terms). Start the first sync now is ticked by default. Add website adds it.
The website then appears on the Websites tab with its status, its domain tier, its last sync and its page count against the page cap. Open it to see its Sync history and Pages, and to use Sync now, Pause, Resume and Delete. Every member can see websites; only admins and owners can change them.
When the preview finds an AI-usage opt-out on a domain you have not verified, a sync would stop at once. If it is your own site, choose Add and verify first: the website is added without a sync (the button reads Add and verify the domain), the portal takes you to the domain verification screen, and once the domain is verified you start the first sync there with Start sync. When robots.txt disallows the start URL, the website cannot be added at all: verification never overrides robots.txt, so allow InfusBot in robots.txt first and check the website again.
With the API
1. Preview the site (optional, but it tells you what a sync will find before any source or page is created):
curl -X POST "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources:preview" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"base_url": "https://docs.example.com/guide/"}'
The answer is 202 with a preview id and status: running. Poll it until status is completed or failed (usually a few seconds):
curl "https://api.infus.ai/v1/knowledge_bases/$KB_ID/source_previews/$PREVIEW_ID" \
-H "Authorization: Bearer $API_KEY"
| Preview field | Meaning |
|---|---|
result.robots | verdict (allowed, disallowed, unavailable_disallow), whether the start URL itself is allowed, and any Crawl-delay |
result.discovery | Whether llms.txt, llms-full.txt and a sitemap were found, and which discovery methods apply |
result.pages_found | In-scope URLs that robots.txt allows, without the URLs skipped for their extension (see How pages are discovered; counted up to 50,000; pages_found_estimated is true when there is no llms.txt or sitemap and the count comes from the start page's links only) |
result.sample_urls | Up to 20 URLs a sync would fetch |
result.robots_blocked_count, robots_blocked_sample | URLs robots.txt keeps InfusBot away from |
result.ai_signal | AI-usage signals found and their effect for your tier (none, warning, blocked) |
result.tier, tier_page_cap, pages_to_index | Your tier, its page cap, and how many pages the first sync would index |
result.bot_protection | true when a bot-protection challenge answered: a sync will be blocked |
failure.code | For a failed preview: https_required (no https answer on port 443), bot_protection, fetch_error or internal_error |
A preview is kept for 1 hour. Previews are limited per organization per hour (429 rate_limited, limit_name: web.previews_per_hour).
2. Add the source. Read the current attestation text version and terms version from GET /v1/knowledge_module → web.attestation.text_version and web.terms.version, then:
curl -X POST "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: add-docs-site-1" \
-d '{
"type": "website",
"base_url": "https://docs.example.com/guide/",
"scope": {"mode": "path", "exclude": ["/guide/archive/**"]},
"attestation": {"accepted": true, "text_version": "web-attestation-2026-09-v1"},
"terms_acceptance": {"accepted": true, "version": "web-terms-2026-09-v1"},
"preview_id": "'"$PREVIEW_ID"'"
}'
attestationis required on every source. Sendaccepted: trueonly after the person adding the source has read and accepted the text; see Responsibility and attestation.terms_acceptanceis required on your organization's first website source, and again after the website-source terms change (web.terms.acceptedis thenfalse).preview_idis optional. With it, a site whose robots.txt disallows the start URL is refused at once (422 robots_disallowed), whateverstart_syncsays. An AI-usage opt-out found on an unverified domain refuses onlystart_sync: true(422 ai_signal_optout): with"start_sync": falsethe source is added and nothing is fetched, so you can verify the domain first and then start the first sync. Withoutpreview_id, the first run finds the same and the source becomesblocked_by_site.start_syncdefaults totrue: the response is201with the queued first run inlast_run. Withfalsethe source is added without a run.- A source added under a domain your organization has already verified (and whose whole scope the verification covers) starts with
tier: verified.
3. Follow the run with GET …/sources/{source_id} (the last_run field), GET …/sources/{source_id}/runs, or the knowledge.source.sync_completed webhook.
Scope
The scope decides which URLs belong to the source. It is checked again on every redirect of a page (a redirect out of scope is not followed). robots.txt and /.well-known/tdmrep.json are the exception: their redirects may lead to another path or host of the same registrable domain (for example example.com → www.example.com), still only over https to public addresses and never to a blocked domain. A robots.txt that redirects to another registrable domain is treated as unreachable, which means everything is disallowed.
| Field | Default | Meaning |
|---|---|---|
mode | path | path: same host, and the URL path starts with path_prefix. host: the whole host. subdomains: the start host without a leading www., and every host under it. |
path_prefix | the start URL's directory | Used by mode: path (/guide/ for https://docs.example.com/guide/intro). |
include, exclude | none | Up to 20 glob patterns each, matched against the normalised path and query. * matches any characters except /, ** matches any characters, ? matches one character except /. The whole path must match, an exclude wins over an include, and regular expressions are not supported. |
max_pages | the tier's page cap | Your own, lower cap (1 to 5,000). A value above the tier cap is stored and applies when the tier allows it; effective_max_pages on the source shows the cap in force. |
max_depth | 5 | Link hops from the start URL, at most web.limits.max_depth (5 by default; a higher value is refused with 422 validation_failed). URLs from llms.txt and sitemaps count as depth 0. |
excluded_urls | none | Up to 1,000 exact URLs that are never requested directly or indexed (content reached through a redirect or a canonical URL that lands on one is discarded). |
Change the scope later with PATCH …/sources/{source_id}: each top-level key you send replaces that key (lists as a whole) and the keys you leave out keep their value. A scope change applies from the next run; pages that fall out of scope are then removed, subject to the deletion hold.
Limits
Platform defaults below; your effective values are in GET /v1/knowledge_module → web.limits. Your platform operator can lower them; they are the same for every organization (plans never change them).
| Limit | Default | When exceeded |
|---|---|---|
| Pages per source, unverified domain | 100 | The run ends completed with stop_reason: page_cap_reached; extra pages are skipped (page_cap) |
| Pages per source, verified domain | 5,000 | Same |
| Sources per knowledge base | 5 | 409 resource_limit_reached (limit_name: max_sources_per_kb) |
Link depth (max_depth) | 5 | Deeper links are not followed |
| Page size (HTML, text, Markdown) | 10 MiB after decompression | Page skipped (size) |
| Linked documents (PDF, DOCX, PPTX, XLSX, CSV) | 50 MB, like uploads | Page skipped (size) |
| Fetching time per run | 4 hours | The run ends completed with stop_reason: time_cap_reached; pages already fetched are processed |
| Previews per organization | 30 per hour | 429 rate_limited |
| Manual syncs of one source | one per 60 minutes | 429 rate_limited with Retry-After |
| Domain verification checks per organization | 30 per clock hour (fixed, not in web.limits) | 429 rate_limited (limit_name: web.verifications_per_hour) |
| Scheduled refresh interval | at least 24 hours, at most 720 | 422 validation_failed |
| URL length | 2,048 characters, at most 20 path segments | Page skipped (trap) |
The page cap counts the pages a source holds (indexed, processing or waiting to be removed), across runs, so a source never grows past it. The same number also caps the pages one run fetches: pages the source already holds are re-checked within that per-run fetch cap. skipped_by_reason.page_cap counts only in-scope URLs beyond the cap. When a run starts while the source holds pages and has room for more (after the cap went up, for example a higher max_pages or a verified domain, or after pages were removed), URLs the source has never seen are fetched first until the room is full or none is left; the URLs it already knows (re-checks, and earlier skips and failures) follow within the rest of the per-run fetch cap. The 1,000 documents per knowledge base limit counts uploaded documents only: website pages count against their source's page cap instead.
A source is unverified until you prove that you control its domain. Domain verification raises the cap to 5,000 pages and allows scheduled refresh.
https only
Website sources use https on port 443 only; plain http (port 80) is never contacted.
- A start URL with
http://is refused with422 https_required;details.suggested_urlholds the same URL withhttps://. http://links found on the site are fetched over https. A host that does not answer https, or a redirect tohttp://, skips the page (https_required).- Other schemes, ports other than 443, IP addresses, user names or passwords in the URL, and internal host names are refused with
422 source_url_rejected(details.rulenames the rule). Every address a host name resolves to must be public, on every request and every redirect, so a site can never point InfusBot at a private network.
Domain verification
Verifying a domain proves that your organization controls the site. It is optional while the platform accepts unverified websites: an unverified website then works, within the smaller limits. When the platform accepts verified websites only, add the website without a sync (start_sync: false; in the portal the first sync is then offered after verification), verify the domain, then sync; a sync of an unverified source is refused with 403 domain_not_verified.
| Unverified domain | Verified domain | |
|---|---|---|
| Pages per source | 100 | 5,000 |
| Syncs | Manual only | Manual and scheduled refresh |
| AI-usage opt-out on the site | Stops the sync (blocked_by_site, ai_signal_optout) | Shown as a warning; the sync continues |
| Request pace on the site (shared by all customers) | About 1 request per second, 2 at a time | Up to 5 requests per second, 4 at a time |
| JavaScript rendering | Not available | Available per source, when the platform has it switched on |
| robots.txt and bot protection | Always obeyed | Always obeyed: verification never changes this |
A verification belongs to your organization and one registrable domain (for example example.com). It is never shared with another organization, and one verification covers every website source of your organization whose whole scope it covers, in any knowledge base. A source added later under a verified domain starts verified.
How to verify
Verify the domain once, in the portal under Settings → Integrations → Domains, or with the platform Domains API. That page explains the three methods (DNS record, file, meta tag) and what makes a check fail.
- Whole scope. The DNS record proves the registrable domain and every host below it. The file and the meta tag prove one host, so they verify a source with
scope.modepathorhoston exactly that host. A source withscope.mode: subdomainsneeds the DNS record. - InfusBot must be able to fetch the file or page. The file and tag checks follow the site's robots.txt (the
InfusBotgroup, else*) and do not get around bot protection. The DNS method sends no request to the site.
The source's verification shows the result:
verification.state | Meaning |
|---|---|
unverified | No verified domain of your organization covers the source's whole scope. |
verified | A verified domain covers it: domain_id, domain, method and verified_at of that domain. The state of each check (pending, failed, lost) is on the Domains page and API. |
A successful check changes the tier only. It never starts a sync and never restarts a stopped source: the portal then offers Start sync (or Resume, for a source an AI-usage opt-out had stopped). With the API, start a sync, or set a blocked_by_site source active first.
Keeping the verification
Leave the record, file or tag in place. The platform checks it again at least once a day and before every scheduled refresh (unless a check succeeded within the last hour). A proof that is gone loses the verification at that check, unless another proof of the domain still holds; a check that cannot decide (the site or DNS does not answer) keeps it for up to 72 hours. See Keeping a domain verified.
When a verification is lost, every source it alone covered drops to the unverified tier at once:
- the page cap is 100 again: nothing is deleted, but a source above 100 pages gains no new page until it is below the cap;
- a switched-on schedule is paused with
schedule.paused_reason: verification_lost; - AI-usage opt-outs stop syncs again;
- a run already under way finishes with the tier it started with.
The organization owner gets one Domain verification lost email per domain and day; no webhook is sent. Verifying the domain again restores the verified tier, and a schedule paused verification_lost resumes by itself.
Scheduled refresh
A source on a verified domain can refresh itself on a schedule. Each scheduled run is an ordinary run (trigger: scheduled) that fetches only what may have changed, so unchanged pages cost nothing.
Turn it on on the website page in the portal (Scheduled refresh → Refresh this website automatically, then How often: every day up to every 30 days), or with the API:
curl -X PATCH "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources/$SOURCE_ID" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"schedule": {"enabled": true, "interval_hours": 24}}'
interval_hoursis at leastweb.limits.min_refresh_hours(24 by default) and at most 720. Without it, the minimum is stored (daily).schedule.next_run_atis the next refresh: one interval after you switch the schedule on or change the interval, plus a small fixed offset per source (under an hour), so schedules switched on together do not all start at once. The run can start a few minutes after that time: at most one scheduled run starts per site at a time, and none while the site already has two runs of any customer under way.- Only a verified source can have a schedule: switching it on or changing it for an unverified source answers
403 domain_not_verified. Switching it off ({"schedule": {"enabled": false}}, or Turn off) always works;next_run_atis thennull. - A scheduled run never overlaps another run of the source. When a run cannot start (the source is paused, held, blocked or syncing, the knowledge base is not
active, billing refuses the work, website sources are switched off), that refresh is skipped without an error andnext_run_atmoves on by whole intervals to the next time in the future (keeping the schedule's rhythm).
What a scheduled run fetches. Before starting, the run confirms the domain verification (see Keeping the verification). An indexed page whose sitemap lastmod has not changed since its content was last confirmed, whose links are known from its last full read, and that was fetched within the last 30 days, is not requested at all (it counts as not_modified). Other pages are normally requested with If-None-Match / If-Modified-Since (pages that are rendered are requested without them), and a page whose text did not change gets no new version and costs nothing. A manual Sync now never skips a page for its sitemap lastmod, so use it to force a check (within the run's page and time caps).
When a schedule pauses
schedule.paused_reason tells you why no scheduled run starts:
paused_reason | Why | How to turn it back on |
|---|---|---|
idle | Nobody searched the knowledge base or used it in chat for 30 days (the platform default), counted from the last query or from when you last switched the schedule on. The source itself is unchanged: its pages stay searchable and Sync now still works. No email is sent. | Turn back on in the portal, or PATCH with {"schedule": {"enabled": true}}. This restarts the 30 days. |
verification_lost | The domain verification was lost. | Verify the domain again: the schedule then resumes by itself. Or turn the schedule off. |
Treat a paused_reason you do not know as "paused".
How pages are discovered
Each run looks for pages in this order, and fetches them in the same order of priority (after the never-seen URLs, when the source has room for new pages; see Limits):
llms.txtat the root of the site: a list of LLM-ready pages that many documentation platforms publish. Its links come first; Markdown versions of pages are preferred when offered. Anllms-full.txtis detected and reported, but not indexed, because it repeats the pages.- Sitemaps: the
Sitemap:lines of robots.txt, then/sitemap.xml(sitemap indexes included). Newer pages (bylastmod) come first. - Links: starting from the start URL, links are followed within the scope, up to the link depth.
When a site has more pages than the cap, the cut follows this order: llms.txt pages first, then sitemap pages, then linked pages. The run reports it (page_cap_reached); nothing is dropped silently. Publishing an llms.txt or a sitemap is the best way to control which pages are indexed first.
Along the way:
- Query strings and fragments are dropped, so tracking and session parameters never create duplicate pages. A
<link rel="canonical">on the same host is honoured. - Links to PDF, DOCX, PPTX, XLSX, CSV, Markdown and text files inside the scope are fetched and processed like uploads of that type. Other content types are skipped (
content_type). - Calendar-like URL patterns, endlessly repeated path segments and similar crawler traps are skipped (
trap). Pages with the same text as another page of the source are kept once (duplicate). - URLs whose path ends in an extension the platform does not index (images, stylesheets, scripts, JSON and XML feeds, fonts, archives, audio, video, other binaries, and document formats other than the ones above, such as
.doc,.xls,.ppt,.odt,.rtfor.epub) are skipped ascontent_typewithout being requested; they get no page entry and are not counted in a preview'spages_found. Any other extension is requested, and the answer's content type decides. - When a Markdown version listed in
llms.txtasks not to be indexed (noindex), it is skipped and the page's HTML version is fetched in its place (the same address without.md;…/index.mdbecomes…/), still subject to the scope, robots.txt and the caps. For the rest of the run, other Markdown versions fromllms.txton that host are replaced by their HTML versions before any request. - Only public pages are fetched: InfusBot sends no stored cookies and no credentials, never logs in and never submits forms. By default it does not run JavaScript, so pages that only show their content after JavaScript runs are indexed with whatever text the server sent; for those, see JavaScript rendering.
JavaScript rendering
Some sites build their content in the browser (single-page apps, client-rendered documentation): the HTML the server sends is an empty shell that scripts fill in. For such a site on a verified domain, InfusBot can render each page in an isolated browser before indexing it.
render | Portal | What happens |
|---|---|---|
off (default) | Off | Pages are indexed from the HTML the site sends. |
auto | Auto | Only pages whose HTML is an empty shell (almost no text and at least one script) are rendered; the others are indexed as sent. |
always | Always | Every HTML page is rendered before indexing. Use it when the content arrives by script even though the page has other text. Slower. |
- Set it on the website page in the portal (JavaScript rendering → Render pages with JavaScript), or with
renderonPOST …/sourcesorPATCH …/sources/{source_id}. The change applies from the next run; a run that is already under way keeps the mode it started with, except that switching rendering off on the platform stops new renders at once, also within a running sync. - When it takes effect. Only while the source's domain is verified and JavaScript rendering is switched on for the platform. It is off by default:
web.render_enabledinGET /v1/knowledge_modulesays whether it is on. On an unverified source, or while the platform has it off, the value is kept but has no effect.render_activeon the source says whether its pages are rendered now. - What is rendered. HTML pages only. robots.txt, sitemaps,
llms.txt, linked documents (PDF and others), previews and verification checks are never rendered. - What the browser loads. The page itself is fetched once, under every rule above (robots.txt, the shared request budget, the signed request). While it renders, the browser may load, over https on port 443 only, scripts, stylesheets and data requests (XHR, fetch) from the page's exact origin (scheme, host and port; only where that origin's robots.txt allows InfusBot) and scripts and stylesheets from other origins. It loads no images, media, fonts or frames, opens no popup or WebSocket, and does not navigate away: a script that navigates to another page ends the render and the HTML the site sent is kept (history changes within a single-page app are fine). These requests carry InfusBot's User-Agent, count against the site's shared request budget and are not signed (see Signed requests). A render has fixed limits on requests, bytes and time; the page is taken as it is when a limit is reached.
- Directives still apply.
noindex,nofollow,noaiand a TDM reservation count whether they are in the HTML the site sent or in the rendered page (the stricter one wins). - When a render fails (or rendering is not possible right now), the page is indexed from the HTML the site sent: a render never fails a page.
- Each run reports
render: the mode in effect for the run and how many pages wererendered,failedorunavailable(not counted: pagesautoleft unrendered). In the portal the sync history shows how many pages were rendered. - No new options.
off,autoandalwaysare the only choices: no browser setting, script, header or wait time can be passed. - Bot protection is still never solved: a page answered by a challenge is not rendered, and the source becomes
blocked_by_site.
InfusBot and robots.txt
InfusBot identifies itself on every request. During a sync the User-Agent is:
Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-<source-id>; +<bot page URL>)
<source-id> is your source's id. Previews and domain verification checks send the same string without the InfusBot-<source-id> part. The source's robots_token field (InfusBot-<source-id>) is what a site owner writes in robots.txt to address only your source; the portal shows it on the source page so you can pass it to the site owner.
InfusBot's public page (/bot on the platform's website; the address is the last part of the User-Agent and web.bot.info_url in GET /v1/knowledge_module) explains to site owners what InfusBot is, how to control it and how to ask for removal. Point a site owner there if they ask about your source.
Rules InfusBot always follows
- robots.txt is always obeyed (RFC 9309). You cannot turn this off, for your own site either: change the site's robots.txt instead.
- The group for
InfusBot-<source-id>applies if the file has one; otherwise theInfusBotgroup; otherwise the*group. The longest matching rule wins, andAllowwins a tie. - A robots.txt that answers 4xx (for example a missing file) means no restrictions. One that answers 5xx, times out or cannot be reached means everything is disallowed: if that happens for the start URL, the source becomes
blocked_by_site(robots_disallowed) until you set itactiveagain and sync. - robots.txt answers are cached for at most 24 hours (at most 10 minutes after a failure), so a change on the site applies to later runs within a day.
Crawl-delayis honoured up to 30 seconds. A delay that makes the page cap unreachable within the run's time limit ends the run withtime_cap_reached.noindex(in<meta name="robots">or theX-Robots-Tagheader) keeps a page out of the index;nofollowstops its links from being followed.- Polite pace. Every site (registrable domain, for example
example.com) has one request budget shared by all customers of the platform: about 1 request per second and at most 2 requests at a time on an unverified domain (up to 5 per second and 4 at a time for sources on a verified domain; the site never sees more than the highest of these). Ten organizations adding the same site do not make ten times the load; they share the budget, so a popular site may sync more slowly.429and503answers withRetry-Afterare respected (up to one hour), and repeated errors pause the site for a while.
How a site owner controls InfusBot
A site owner uses ordinary robots.txt groups. These are three alternatives, not one file: groups for the same user agent are combined, so pick the one that matches the policy you want.
Block InfusBot for every customer:
User-agent: InfusBot
Disallow: /
Allow only the documentation:
User-agent: InfusBot
Allow: /docs
Disallow: /
Block one source only (other customers keep the InfusBot or * rules):
User-agent: InfusBot-5a4b3c2d-1e0f-4a9b-8c7d-6e5f4a3b2c1d
Disallow: /
If you manage the site yourself and your robots.txt blocks crawlers in general, add an InfusBot group (or one for your source's token) that allows the pages you want indexed. InfusBot then follows only that group, not *. If you verify the domain with the verification file, allow that path too:
User-agent: *
Disallow: /
User-agent: InfusBot
Allow: /docs/
Allow: /.well-known/infus-verification
Disallow: /
Letting InfusBot through a firewall, CDN or bot protection
robots.txt is not the only gate. A web application firewall, a CDN's bot management or a login wall can refuse InfusBot, and InfusBot never works around them (the source becomes blocked_by_site). If the site is yours, allow InfusBot there. Its name does not depend on the platform's brand: the product token InfusBot and the verification name infus-verification are always the same. It can be recognized in three ways; combine them, because a user agent alone is easy to copy.
1. User agent. Every request carries a User-Agent that contains InfusBot/1.0:
Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-<source-id>; +https://<platform-website>/bot)
Previews and verification checks send it without the InfusBot-<source-id> part. Match on InfusBot/ rather than the whole string, so a version change does not break your rule. The address at the end is InfusBot's public page (web.bot.info_url in GET /v1/knowledge_module).
2. Published IP addresses. The addresses InfusBot sends from are published as JSON next to the bot page, at https://<platform-website>/bot/ips.json, in the format other crawlers use (the address below is a documentation example):
{ "creationTime": "2026-09-28T07:46:31Z", "prefixes": [ { "ipv4Prefix": "192.0.2.10/32" } ] }
- Only InfusBot sends from these addresses, and it sends from no other address.
- A new address is listed at least 30 days before InfusBot starts using it, so an allowlist that follows the file is always ahead. An address InfusBot stops using stays listed for another 7 days. The exception is an emergency (an address lost or revoked): the replacement is then published and used at once, with notice the same day, and allowlists that only know the old address need updating.
- Re-read the file at least once a day (it is cached for at most a day) instead of copying the addresses once.
- InfusBot's production traffic never comes from an address that is not on the list. An empty
prefixeslist means no production address is in use yet; until one is listed, rely on the user agent and the signatures below.
3. Signed requests (Web Bot Auth). Every request InfusBot makes itself, including robots.txt, verification checks and each redirect, is signed with an HTTP Message Signature (RFC 9421) following the IETF Web Bot Auth draft (draft-ietf-webbotauth-httpsig-protocol). Each request carries three headers:
Signature-Agent: "https://<platform-website>"
Signature-Input: sig1=("@method" "@authority" "@path" "signature-agent");created=1790610300;keyid="<key id>";alg="ed25519";expires=1790610360;nonce="<random>";tag="web-bot-auth"
Signature: sig1=:<signature>:
- The public keys are in the key directory
https://<platform-website>/.well-known/http-message-signatures-directory(media typeapplication/http-message-signatures-directory+json; Ed25519 keys as JWKs,keyid= the key's JWK SHA-256 thumbprint). The directory response is itself signed with the same keys (taghttp-message-signatures-directory). - A signature is valid for a short time (60 seconds by default) and covers the method, the host and the path (not the query string), so it cannot be reused for another host or path; each signature also carries a random
nonce, which a verifier can use to reject a repeated signature within its validity. Keys are rotated with an overlap: a new key appears in the directory at least a day before it signs, so a verifier that caches the directory for up to a day keeps working. Only a compromised key is replaced at once. - The scripts, stylesheets and data a page loads while it is rendered are requested with the same User-Agent and from the same published addresses, but they are not signed. A rule that requires a valid signature must therefore not apply to these requests, or JavaScript rendering of your site loses them.
- CDNs and bot-management products that verify Web Bot Auth can use the directory to recognize InfusBot. Whether a particular CDN lists InfusBot as a verified bot depends on that CDN's own bot directory; if yours does not, allow InfusBot with a rule as below.
Example: Cloudflare. Create a WAF custom rule with the Skip action and an expression such as:
(http.user_agent contains "InfusBot/" and ip.src in {192.0.2.10})
using the addresses from ips.json (while the list is empty, the user-agent condition alone lets anyone who copies the user agent through, so limit such a rule to the paths you want indexed). Choose which security features the rule skips, for example the remaining custom rules, rate limiting rules, managed rules and, where your plan has it, Super Bot Fight Mode. The free plan's Bot Fight Mode cannot be skipped by a rule. A JavaScript challenge or "I'm Under Attack" mode stops InfusBot wherever it applies, because InfusBot never solves challenges (also not when a page is rendered), so keep InfusBot's requests out of it. Other WAFs and CDNs have equivalent allow or bypass rules keyed on user agent and source address.
After changing the site's settings, set the source active again and start a sync.
AI-usage signals
Some sites state that their content must not be used by AI systems. InfusBot looks for these signals:
| Signal | Where | Counts as an opt-out when |
|---|---|---|
| Content Signals | a Content-Signal: line in robots.txt | ai-input=no |
| IETF AI preferences | the Content-Usage header or robots.txt | the category covering AI use says no |
| TDMRep | /.well-known/tdmrep.json, or the tdm-reservation: 1 header or meta tag | a reservation applies to the page |
noai | <meta name="robots"> or X-Robots-Tag | always (noimageai for image-only documents) |
ai-train=no or search=no alone are recorded but are not an opt-out: the platform does not train models on your pages or run a search engine.
On an unverified domain any opt-out stops the sync: a site-wide signal (in robots.txt or tdmrep.json) as soon as those two files are read, before llms.txt, sitemaps or any page is requested; a page-level signal as soon as it is found (that page is not indexed). The source becomes blocked_by_site with status_reason: ai_signal_optout; pages indexed earlier stay until you delete them. On a verified domain the signal is shown as a warning and the sync continues. The findings are on the source (ai_signal) and on each run (ai_signals). A source added with the preview_id of a completed preview carries that preview's findings in ai_signal right away (judged for the new source's tier), so a website added with Add and verify first shows the signal before its first sync; the first run replaces them.
When a site says no: blocked_by_site
InfusBot never gets around bot protection. It does not retry with other headers, use proxies, rotate IP addresses or user agents, solve CAPTCHAs, replay cookies or pretend to be a browser. When a site refuses, the run stops and the source becomes blocked_by_site:
status_reason | What happened |
|---|---|
bot_protection | A bot-protection challenge (for example a Cloudflare, Turnstile, hCaptcha or reCAPTCHA page) answered InfusBot. |
access_denied | The start URL answered 401 or 403 on consecutive runs. |
robots_disallowed | robots.txt disallows the start URL for your source, or robots.txt answered with a server error or could not be reached. |
ai_signal_optout | The site (or a page) of an unverified domain carries an AI-usage opt-out. |
There are three ways out:
- Let InfusBot in. If you control the site, allow InfusBot in its robots.txt and in its bot-protection or firewall settings: see Letting InfusBot through a firewall, CDN or bot protection for the user agent, the published IP addresses and the signed requests. For an AI-usage opt-out on your own site, remove the opt-out, or verify the domain: the signal then becomes a warning. Verification does not change robots.txt or bot protection: those still have to allow InfusBot.
- Publish an
llms.txtor a sitemap that lists the pages you want indexed, so discovery does not depend on crawling pages the protection guards. - Upload the content as files. Export the documentation (HTML, Markdown, PDF) and upload it to the knowledge base.
After fixing the site, set the source active again (PATCH …/sources/{source_id} with {"status": "active"}, or resume it on the source page in the portal) and start a sync. The organization owner gets an email when a source is blocked, and a knowledge.source.blocked webhook is sent.
Sync runs
A run is one pass over the site. Start one with POST …/sources/{source_id}:sync, or with Sync now on the source page in the portal; on a verified domain a scheduled refresh starts runs too. A run's trigger is manual or scheduled. Only one run per source can be active; starting another returns 409 source_sync_in_progress, and a new manual sync of the same source is possible 60 minutes after the previous one started (429 rate_limited before that).
| Run status | Meaning |
|---|---|
queued, running | Discovering and fetching |
awaiting_budget | Page processing is waiting for a paid allowance (see Pricing); no new pages are fetched meanwhile. After 24 hours without an allowance the run fails with budget_exhausted. |
completed | Finished. stop_reason says whether a cap cut it short. |
held | Finished, but its removals wait for your decision (see the deletion hold). |
stopped | The site refused (the source is now blocked_by_site), the domain was blocked or website sources were switched off, or the knowledge base stopped being active. |
failed | The site kept failing, billing refused the work, or a platform error. |
cancelled | You paused or deleted the source, or a platform operator cancelled the run. |
Each run reports what it found in counts: discovered, in_scope, fetched, not_modified, unchanged, new, changed, removed, pending_removal, failed, skipped and skipped_by_reason. It also records what robots.txt said (robots), the AI-usage findings (ai_signals), how long it waited for the site's shared budget (budget_wait_seconds, host_pauses) and, when it rendered pages, the JavaScript rendering summary (render). In the portal, a source's runs are on its Sync history tab.
stop_reason | Run status | Meaning |
|---|---|---|
null | completed | Everything discovered within the caps was processed. |
page_cap_reached | completed | The source's page cap was reached; the remaining URLs were not fetched. |
time_cap_reached | completed | The run's fetching time ran out; pages already fetched were processed. |
nothing_discovered | completed | No in-scope URL that robots.txt allows was found. |
deletion_threshold | held | Removals above the threshold are waiting; additions and changes were published. |
bot_protection, access_denied, robots_disallowed, ai_signal_optout | stopped | The site said no; see blocked_by_site. |
domain_blocked, tenant_disabled, web_disabled | stopped | The domain is blocked on the platform, website sources were switched off for your organization, or switched off on the platform. |
kb_unavailable | stopped | The knowledge base stopped being active during the run (for example suspended or frozen). |
host_paused | failed | The site kept answering 429, 503 or server errors, or timing out, and the resulting pause (from Retry-After, or after repeated errors) would end after the run's fetching time limit. Shorter pauses are waited out. |
budget_exhausted | failed | Pages waited 24 hours for a paid allowance. |
billing_blocked | failed | Billing refused a page with payment_suspended; no further pages are fetched. The refused page is not charged; pages accepted before the refusal are processed and billed normally. |
internal_error | failed | Platform failure after retries; unprocessed pages are not charged. |
source_paused, source_deleted, operator_cancelled | cancelled | You paused or deleted the source, or a platform operator cancelled the run. |
Treat a stop_reason or status_reason you do not know as "other": new values may be added.
Every page is a document
Each page becomes one document of the knowledge base, with origin: web:
- The document
nameis the page title (or its URL path);sourceholds its provenance:{source_id, url, fetched_at, verified}. List a source's pages as documents withGET …/documents?source_id=<source_id>(or all website pages with?origin=web). - Search hits and chat citations of a website page carry the same
source, so you can link the answer to the page. Labelverified: falseas third-party content. - A changed page gets a new version: the old version keeps answering until the new one is ready, exactly like a file replacement. A file replacement (
PUT …/file) of a website page is refused (409 invalid_document_state): new versions come from the site. - Common kinds of hidden content (scripts, elements hidden with the
hiddenattribute,aria-hiddenor inline styles such asdisplay:none) and invisible characters (zero-width, bidirectional control and tag characters) are removed before indexing; the document then carries thehidden_content_removedwarning. Page text is passed to the model as untrusted reference material, with the instruction not to follow commands in it. As with uploads, this reduces but cannot fully eliminate the risk that a page crafted to manipulate a model changes an answer (prompt injection), so index sites you trust. - A page is unchanged when the site answers
304 Not Modified, or when its extracted text is identical to the indexed version. An unchanged page gets no new version, sends no webhook and costs nothing.
Pages and skip reasons
GET …/sources/{source_id}/pages lists the pages of a source with their status (queued, fetching, processing, indexed, skipped, failed, pending_removal, removed, excluded), document_id, http_status, depth, how it was found (discovered_via) and when it last changed. Filter with ?status= and ?skip_reason=.
skip_reason | Meaning |
|---|---|
robots | robots.txt disallows the URL. |
noindex | The page asks not to be indexed. |
ai_signal | Page-level AI-usage opt-out on an unverified domain (the run stops too). |
content_type | Not an accepted type, or the file signature disagrees. Links with an obvious non-document extension (images, stylesheets, scripts, feeds, archives and similar) get this reason without being requested. |
size | Larger than the page or file size limit, or suspiciously compressed. |
scope | Outside the source's scope. |
excluded | Matches an exclude pattern, excluded_urls or an excluded page; also when a redirect or canonical URL lands on one. |
redirect_out_of_scope | A redirect left the scope and was not followed. |
https_required | A redirect to http://, or the host does not answer https on port 443. |
url_rejected | The URL leads to an address the platform never contacts (private or internal networks). |
trap | The URL looks like a crawler trap (too long, too many or repeated segments). |
duplicate | Same text as another page of this source. |
page_cap | Found and in scope, but beyond the page cap. |
http_error | The site answered another 4xx or 5xx status. |
fetch_error | DNS, connection or TLS failure, a timeout, a very slow answer, or too many redirects. |
bot_protection | A challenge page answered this URL (the source is then blocked_by_site). |
A page that answered 404 or 410 after being indexed is not a skip: it is a removal.
Removals and the deletion hold
A page indexed by an earlier run is removed when a run finds it gone (404/410), disallowed by robots.txt, marked noindex, out of scope after a scope change, or excluded. A page that simply was not found again is removed only after a complete discovery: a run cut short by a cap, a stop or failed fetches never removes pages just because it did not reach them. Removing a page deletes its document, with the usual knowledge.document.deleted webhook.
To protect you from a broken sitemap or a site outage wiping the knowledge base, a run that would remove more than 20 % (the platform default; the run reports the threshold in force as deletion.threshold_percent) of the pages that were indexed before the run is held (exactly 20 % is not held):
- the run ends
held(stop_reason: deletion_threshold) and the source becomesheld; - additions and changes are published normally; the pages to remove stay searchable, with page status
pending_removal; - the organization owner gets an email, and
knowledge.source.sync_completedcarries the run; - no new run starts until you decide, and the hold never expires on its own.
Decide with:
curl -X POST "https://api.infus.ai/v1/knowledge_bases/$KB_ID/sources/$SOURCE_ID/runs/$RUN_ID:resolve" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"decision": "apply_deletions"}'
| Decision | Effect |
|---|---|
apply_deletions | The held pages' documents are deleted (pages you deleted meanwhile are skipped). |
discard_deletions | The pages stay indexed; the next run looks at them again. |
A decision is recorded once on the run (deletion.decision, deletion.decided_at); sending the same decision again returns 200, a different one returns 409 invalid_source_state. The run keeps the status held (and its counts as they were when it ended): a non-null deletion.decision is what marks it as resolved. A held source becomes active; a paused source stays paused. In the portal a held source shows Needs review and a Review page removals notice with Remove pages (apply_deletions) and Keep pages (discard_deletions); the pages stay searchable until you decide.
Excluded pages
You can delete a website page's document like any document (DELETE …/documents/{document_id} or a batch delete). Its page then becomes excluded: no later run requests it directly or indexes it again, even if the site still links to it. If a redirect or a canonical URL leads to an excluded page, the fetched content is discarded. There is no limit on the number of excluded pages.
- List them with
GET …/sources/{source_id}/pages?status=excluded. - Bring pages back with
PATCH …/sources/{source_id}and{"reinclude_urls": ["https://docs.example.com/guide/old-page/"]}(up to 100 URLs per call); they are eligible again from the next run. In the portal, open the website's Pages tab, filter by Excluded, select the pages and choose Re-include selected. - To keep URLs out before they are ever indexed, use
scope.excludepatterns orscope.excluded_urlsinstead.
Source status
| Status | Portal | Meaning | What you can do |
|---|---|---|---|
active | Active | Idle and ready | Sync, pause, change, delete |
syncing | Syncing | A run is queued, running or waiting for budget | Pause (cancels the run), delete |
held | Needs review | The latest run's removals wait for a decision (status_reason: deletion_threshold) | Resolve the run; pause; delete |
blocked_by_site | Blocked by site | The site refused (see above) | Fix the cause, set active, sync |
paused | Paused | Paused by you (customer); or left paused when the platform lifted a block: rarely unblocked (domain block lifted) or tenant_reenabled (website sources switched back on for your organization), or takedown_lifted (the block ended after a removal request deleted the source's pages) | Set active |
disabled_by_admin | Disabled | The platform blocked the domain (domain_blocked) or switched website sources off for your organization (tenant_disabled) | Contact support; pages stay searchable, and you can still delete |
deleting | Deleting | The source and its pages are being deleted | Nothing |
When neither a domain block nor the organization-wide switch-off applies any more (the platform lifted the block, or switched website sources back on for your organization), each source gets back the status it had before (active, also for a source that was syncing; paused with its reason; blocked_by_site with the site's reason; held while its removals still wait for a decision, otherwise active). No run starts, and no webhook or email is sent. Only a source whose earlier status is not known is left paused, with status_reason unblocked (domain block lifted) or tenant_reenabled (website sources switched back on): resume it when you are ready.
After a removal request that deleted pages, a source does not get its earlier status back. If the platform deleted a source's indexed pages because of a removal request (takedown) during the block, the source stays paused with status_reason: takedown_lifted when the block is lifted, even if it was active before, and has no pages. Make sure you may use the content again before you resume it with {"status": "active"}: the next sync fetches the pages again. A removal request that only blocked the domain, without deleting pages, keeps the normal behaviour above. The portal shows: "The platform deleted this website's pages after a takedown request and has since ended the block. Make sure you may use the content before you resume the website: the next sync fetches the pages again."
Pause and resume with PATCH …/sources/{source_id} and {"status": "paused"} or {"status": "active"}. Pausing cancels a running sync; indexed pages stay searchable while paused. Resuming a source whose held run is still unresolved returns it to held. Other status changes are refused with 409 invalid_source_state.
Deleting a source (DELETE …/sources/{source_id}, 202) cancels its run and deletes every page document through the normal deletion process, with one knowledge.document.deleted webhook per page. Deleting the knowledge base deletes its sources.
Pricing
Website sources use the same meters as uploads (Pricing and Billing):
- No crawl fee. Previews, discovery, robots.txt and sitemap reads and page fetches are free.
- Processing tokens for each new or changed page, and storage for the page's original bytes (the fetched HTML, the rendered HTML when the page was rendered, or the linked file), exactly as for an uploaded file. Free-trial allowances apply.
- Unchanged pages are never charged again, and neither are platform retries. JavaScript rendering adds no charge: a rendered page is billed like any other page, and not at all when its text did not change. A scheduled refresh costs the same as a manual sync: only new and changed pages are billed.
- When processing cannot be paid (trial used up, no balance), pages wait in
awaiting_budgetand the run waits with them for up to 24 hours.
Responsibility and attestation
The summary below is a plain-language overview of how responsibility is shared. It is not legal advice, and the binding text is the platform's terms of use and acceptable-use policy (the address is in web.terms.url).
- You attest, for every source. Adding a website requires an explicit confirmation, shown next to a checkbox in the portal and required as
attestationin the API. The text is versioned;web.attestation.textalways has the current one. At the time of writing it reads: "I confirm that I am authorised to index the content of this website. I am responsible for this content and for how it is used, and I will indemnify the platform against third-party claims arising from it." - You accept the website-source terms on your organization's first source and after they change. They forbid, among other things, indexing sites you are not authorized to index, and infringing copyright or data-protection law through the indexed content.
- What is recorded. For each source the platform records who accepted, when, from which IP address and which text version. The IP address is kept as evidence and never returned by the API.
- The platform's own care. InfusBot identifies itself, always obeys robots.txt, never gets around bot protection, shares one polite request budget per site across all customers, stops at AI-usage opt-outs on unverified domains and fetches only public pages. Each run records what robots.txt said and what signals were found.
- Site owners can object. A site owner can block InfusBot or a single source in robots.txt, or ask the platform to remove their content (contact details are on the bot page). On a valid request the platform blocks the domain for every customer: sources whose start URL is on that domain become
disabled_by_adminwithstatus_reason: domain_blocked, while sources with a wider scope that merely includes the domain keep running without it. If the owner asks, the indexed pages on that domain are deleted, in whichever source they are. You are notified by email and with aknowledge.source.blockedwebhook. The platform can also switch website sources off for an organization that misuses them (tenant_disabled). When a block is lifted, sources get back the status they had before, except sources whose pages were deleted on the owner's request: those stay paused (takedown_lifted) until you resume them (see Source status).
Webhooks and emails
Two webhook events cover website sources, in addition to the usual knowledge.document.* events for each page (which carry origin: web and the source_id):
| Event | Sent when |
|---|---|
knowledge.source.sync_completed | A run ended, whatever the result: completed, held, stopped, failed or cancelled. Exactly one per run; read data.run.status and data.run.stop_reason. |
knowledge.source.blocked | A source became blocked_by_site or disabled_by_admin, with or without a run (for example after a removal request). |
The organization owner also gets an email when a source is blocked, when a run fails, when a run is held for a deletion decision, and when a domain verification is lost. Losing a verification sends no webhook: read verification and tier on the source.
Errors
The codes specific to website sources are listed in Error Codes: web_sources_disabled, domain_blocked, domain_not_verified, source_sync_in_progress, invalid_source_state, duplicate_source, https_required, source_url_rejected, robots_disallowed, ai_signal_optout, attestation_required and terms_acceptance_required. Verification checks are limited with 429 rate_limited (limit_name: web.verifications_per_hour), and previews and syncs also have a short per-minute burst limit per organization (limit_name: gateway.web_previews_per_minute, gateway.web_syncs_per_minute).