InfusBot is the web crawler behind our knowledge base feature. If you found it in your server logs, this page explains why it visited, how to recognize it, and how to control or stop it.
Our customers build knowledge bases for their own AI assistants. One way to fill a knowledge base is to add a website — typically the customer's own product documentation or help center. InfusBot fetches the public pages of that website, on that customer's request, so they can be indexed into the customer's knowledge base.
InfusBot only crawls sites a customer has explicitly added, and only within the scope they chose. It is not a general-purpose web crawler and does not build a public search index. It does not log in, submit forms, or fetch pages that require an account.
Every crawl request carries a User-Agent header of this form, where <source-id> is the ID of the website source that made the request (a UUID, in lower case):
Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-<source-id>; +https://dev.infus.ai/bot)For example:
Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-3f6c2a9e-8b1d-4e7a-9c55-0d2e4b6a8f10; +https://dev.infus.ai/bot)A customer's preview of a site before adding it, and domain-verification checks (including the regular re-checks of a verified domain), carry no source ID. They send it without the source part and follow the InfusBot group of your robots.txt, then the * group:
Mozilla/5.0 (compatible; InfusBot/1.0; +https://dev.infus.ai/bot)To match InfusBot in a log filter or firewall rule, look for "InfusBot/" rather than the whole string, so a version change does not break your rule. A user agent is easy to copy, so combine it with the IP addresses (section 5) or the request signatures (section 6).
InfusBot answers to two robots.txt product tokens:
InfusBot — every InfusBot request, for every customer.InfusBot-<source-id> — one website source. Each website a customer adds gets its own ID, so you can allow or block a single source without affecting anyone else.When a robots.txt has a group for InfusBot-<source-id>, that source follows it; otherwise it follows the InfusBot group, and if there is none, the * group.
noindex and nofollow in the X-Robots-Tag header and in a robots meta tag (also one named for InfusBot).ai-input=no), the IETF AI preferences Content-Usage header or robots.txt rule (an AI-use category set to n), TDMRep, or noai in the X-Robots-Tag header or a robots meta tag. Training-only preferences are recorded but do not stop it (the content is not used for training). Verification belongs to one customer: only for that customer's sources are these signals shown as a warning instead. Pages indexed before an opt-out appeared stay until removed (section 7).429 or 503 it pauses your domain for the Retry-After time (at most 60 minutes), or without one for 60 seconds, doubling up to 15 minutes; five errors in a row pause it for 15 minutes.Off by default. Only for a customer that has verified your domain with us, and only if that customer turned JavaScript rendering on for the website, InfusBot can run a page's scripts in an isolated headless browser after fetching the page as described above. The browser never requests the page again. From the page's own origin (same scheme, host and port) it loads scripts, stylesheets and data requests (fetch or XHR) that your robots.txt allows for that source. From any other origin, including your other subdomains and CDNs, it loads only scripts and stylesheets, without checking that origin's robots.txt. Images, fonts, media, frames and further navigation are never loaded. These requests carry the same User-Agent, count toward the same per-domain budget and come from the same addresses (section 5), but they are not signed (section 6).
Add one of these groups to the robots.txt at the root of your site:
User-agent: InfusBot
Disallow: /InfusBot stops fetching every page on this host, for every customer. A group for a single source (InfusBot-<source-id>) still takes precedence for that source, so remove or change any such group too.
User-agent: InfusBot
Allow: /docs$
Allow: /docs/
Disallow: /The longest matching rule wins (RFC 9309), so /docs and everything under /docs/ may be fetched and the rest of the site may not (also not /docs-private: $ ends a rule).
User-agent: InfusBot-3f6c2a9e-8b1d-4e7a-9c55-0d2e4b6a8f10
Disallow: /Copy the source ID from the User-Agent in your logs. Only that source is blocked; other customers' sources on the same site keep the rules of the InfusBot (or *) group.
To target one source, copy its ID from the User-Agent in your server logs and write the group as InfusBot-<source-id>. Matching is case-insensitive, and a group for one source replaces the InfusBot and * groups for that source only.
The addresses InfusBot sends its requests from are published as JSON at /bot/ips.json, in the same format other crawlers use (creationTime and a list of ipv4Prefix / ipv6Prefix entries). Read it from your firewall or CDN at least once a day rather than copying the addresses once: a new address is listed at least 30 days before InfusBot starts using it, and an address it stops using stays listed for another 7 days. Only if an address is lost suddenly is its replacement published and used at once (with notice the same day), and the lost address removed; a copied list then stops matching until you refresh it.
| Prefix | Family |
|---|---|
| 212.103.39.4/32 | IPv4 |
This is a development environment: its address is shared with other workloads and hosts, so it is not proof of InfusBot on its own. Check the request signatures (section 6).
For step-by-step help with a web application firewall, CDN or bot protection, see letting InfusBot through a firewall, CDN or bot protection in our documentation.
InfusBot signs every request it sends itself, including robots.txt fetches, verification checks and each redirect, with an HTTP Message Signature (RFC 9421) following the IETF Web Bot Auth protocol. A user agent can be copied by anyone; a valid signature can only come from InfusBot. The one exception is the browser that renders a page's JavaScript (section 3): its script, stylesheet and data requests are not signed. Each signed request carries three headers:
Signature-Agent: "https://dev.infus.ai"
Signature-Input: sig1=("@method" "@authority" "@path" "signature-agent");created=…;keyid="…";alg="ed25519";expires=…;nonce="…";tag="web-bot-auth"
Signature: sig1=:…:Signature-Agent names where the public keys are published. The key directory is at:
https://dev.infus.ai/.well-known/http-message-signatures-directoryIt is served as application/http-message-signatures-directory+json: a JSON Web Key Set of Ed25519 keys, whose keyid is the key's JWK SHA-256 thumbprint. The directory response is itself signed with the same keys (tag http-message-signatures-directory). Open the key directory.
Signature-Agent is exactly "https://dev.infus.ai". Anyone can sign with their own key and name their own origin, so reject any other value.Content-Digest and directory signature must verify for that origin.keyid in Signature-Input.Signature-Agent header, and check the Ed25519 signature.expires has passed (signatures are valid for about 60 seconds) or the tag is not web-bot-auth.CDNs and bot-management products that support Web Bot Auth do these steps for you. Keys are rotated with an overlap: a new key appears in the directory at least a day before it signs anything, and an old key stays listed for a day after its last signature. A key that may have been exposed is replaced at once and removed from the directory; there is no revocation, so a copy of the directory you cached still trusts it until you refresh.
Blocking InfusBot in robots.txt is the fastest way to stop new crawling. If you also want content from your site removed from our customers' knowledge bases, or you believe InfusBot is misbehaving, contact us. Please include:
On a valid request we block the domain for all customers, stop every source that crawls it and, if you ask, delete the pages already indexed from it.
Use the form below to prepare a request: it checks the details and then hands the request to your email app, addressed to info@infus.ai. You can also write to that address directly.