← Back to Home

InfusBot

InfusBot is the web crawler behind our knowledge base feature. If you found it in your server logs, this page explains why it visited, how to recognize it, and how to control or stop it.

1. Why InfusBot visits your site

Our customers build knowledge bases for their own AI assistants. One way to fill a knowledge base is to add a website — typically the customer's own product documentation or help center. InfusBot fetches the public pages of that website, on that customer's request, so they can be indexed into the customer's knowledge base.

InfusBot only crawls sites a customer has explicitly added, and only within the scope they chose. It is not a general-purpose web crawler and does not build a public search index. It does not log in, submit forms, or fetch pages that require an account.

2. How to recognize InfusBot

Every crawl request carries a User-Agent header of this form, where <source-id> is the ID of the website source that made the request (a UUID, in lower case):

Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-<source-id>; +https://dev.infus.ai/bot)

For example:

Mozilla/5.0 (compatible; InfusBot/1.0; InfusBot-3f6c2a9e-8b1d-4e7a-9c55-0d2e4b6a8f10; +https://dev.infus.ai/bot)

A customer's preview of a site before adding it, and domain-verification checks (including the regular re-checks of a verified domain), carry no source ID. They send it without the source part and follow the InfusBot group of your robots.txt, then the * group:

Mozilla/5.0 (compatible; InfusBot/1.0; +https://dev.infus.ai/bot)

To match InfusBot in a log filter or firewall rule, look for "InfusBot/" rather than the whole string, so a version change does not break your rule. A user agent is easy to copy, so combine it with the IP addresses (section 5) or the request signatures (section 6).

InfusBot answers to two robots.txt product tokens:

  • InfusBot — every InfusBot request, for every customer.
  • InfusBot-<source-id> — one website source. Each website a customer adds gets its own ID, so you can allow or block a single source without affecting anyone else.

When a robots.txt has a group for InfusBot-<source-id>, that source follows it; otherwise it follows the InfusBot group, and if there is none, the * group.

3. The rules InfusBot always follows

  • It always obeys robots.txt (RFC 9309). Our customers cannot turn this off. If robots.txt answers with a server error (5xx), times out or cannot be reached, InfusBot treats the whole site as disallowed. So does a robots.txt that redirects to another registrable domain (a redirect within your domain, for example to www, is followed). A missing robots.txt (4xx) means no restrictions.
  • robots.txt is re-read at least every 24 hours (after a failure, within 10 minutes), so a change you make takes effect within 24 hours. Crawl-delay is honored, up to 30 seconds.
  • It honors noindex and nofollow in the X-Robots-Tag header and in a robots meta tag (also one named for InfusBot).
  • For a customer that has not verified your domain with us, an AI-usage opt-out stops that customer's crawl: Content Signals (ai-input=no), the IETF AI preferences Content-Usage header or robots.txt rule (an AI-use category set to n), TDMRep, or noai in the X-Robots-Tag header or a robots meta tag. Training-only preferences are recorded but do not stop it (the content is not used for training). Verification belongs to one customer: only for that customer's sources are these signals shown as a warning instead. Pages indexed before an opt-out appeared stay until removed (section 7).
  • It is gentle. Each registrable domain (for example example.com with all its subdomains) has one request budget, shared by all of our customers: by default about 1 request per second and at most 2 requests at a time, however many customers have added your site. A customer that verified the domain may use up to 5 requests per second and 4 at a time; Crawl-delay slows both.
  • It backs off. On 429 or 503 it pauses your domain for the Retry-After time (at most 60 minutes), or without one for 60 seconds, doubling up to 15 minutes; five errors in a row pause it for 15 minutes.
  • It never gets around bot protection. If your site shows a challenge or blocks the request, InfusBot stops and reports the site as blocked to our customer. It does not use proxy networks to hide where it comes from, rotate IP addresses, change its user agent, solve CAPTCHAs, or disguise itself as a browser.
  • It uses HTTPS only (port 443). Sites that are only available over plain HTTP are not crawled.

Pages that need JavaScript

Off by default. Only for a customer that has verified your domain with us, and only if that customer turned JavaScript rendering on for the website, InfusBot can run a page's scripts in an isolated headless browser after fetching the page as described above. The browser never requests the page again. From the page's own origin (same scheme, host and port) it loads scripts, stylesheets and data requests (fetch or XHR) that your robots.txt allows for that source. From any other origin, including your other subdomains and CDNs, it loads only scripts and stylesheets, without checking that origin's robots.txt. Images, fonts, media, frames and further navigation are never loaded. These requests carry the same User-Agent, count toward the same per-domain budget and come from the same addresses (section 5), but they are not signed (section 6).

4. Controlling InfusBot with robots.txt

Add one of these groups to the robots.txt at the root of your site:

Block InfusBot entirely

User-agent: InfusBot
Disallow: /

InfusBot stops fetching every page on this host, for every customer. A group for a single source (InfusBot-<source-id>) still takes precedence for that source, so remove or change any such group too.

Allow only your documentation

User-agent: InfusBot
Allow: /docs$
Allow: /docs/
Disallow: /

The longest matching rule wins (RFC 9309), so /docs and everything under /docs/ may be fetched and the rest of the site may not (also not /docs-private: $ ends a rule).

Block a single source

User-agent: InfusBot-3f6c2a9e-8b1d-4e7a-9c55-0d2e4b6a8f10
Disallow: /

Copy the source ID from the User-Agent in your logs. Only that source is blocked; other customers' sources on the same site keep the rules of the InfusBot (or *) group.

To target one source, copy its ID from the User-Agent in your server logs and write the group as InfusBot-<source-id>. Matching is case-insensitive, and a group for one source replaces the InfusBot and * groups for that source only.

5. IP addresses

The addresses InfusBot sends its requests from are published as JSON at /bot/ips.json, in the same format other crawlers use (creationTime and a list of ipv4Prefix / ipv6Prefix entries). Read it from your firewall or CDN at least once a day rather than copying the addresses once: a new address is listed at least 30 days before InfusBot starts using it, and an address it stops using stays listed for another 7 days. Only if an address is lost suddenly is its replacement published and used at once (with notice the same day), and the lost address removed; a copied list then stops matching until you refresh it.

Published addresses, as of 2026-09-28T07:46:31Z
PrefixFamily
212.103.39.4/32IPv4

This is a development environment: its address is shared with other workloads and hosts, so it is not proof of InfusBot on its own. Check the request signatures (section 6).

For step-by-step help with a web application firewall, CDN or bot protection, see letting InfusBot through a firewall, CDN or bot protection in our documentation.

6. Signed requests (Web Bot Auth)

InfusBot signs every request it sends itself, including robots.txt fetches, verification checks and each redirect, with an HTTP Message Signature (RFC 9421) following the IETF Web Bot Auth protocol. A user agent can be copied by anyone; a valid signature can only come from InfusBot. The one exception is the browser that renders a page's JavaScript (section 3): its script, stylesheet and data requests are not signed. Each signed request carries three headers:

Signature-Agent: "https://dev.infus.ai"
Signature-Input: sig1=("@method" "@authority" "@path" "signature-agent");created=…;keyid="…";alg="ed25519";expires=…;nonce="…";tag="web-bot-auth"
Signature: sig1=:…:

Signature-Agent names where the public keys are published. The key directory is at:

https://dev.infus.ai/.well-known/http-message-signatures-directory

It is served as application/http-message-signatures-directory+json: a JSON Web Key Set of Ed25519 keys, whose keyid is the key's JWK SHA-256 thumbprint. The directory response is itself signed with the same keys (tag http-message-signatures-directory). Open the key directory.

How to verify a request

  1. Check that Signature-Agent is exactly "https://dev.infus.ai". Anyone can sign with their own key and name their own origin, so reject any other value.
  2. Fetch that origin's key directory over HTTPS (cache it for up to a day). It is signed too: its Content-Digest and directory signature must verify for that origin.
  3. Find the key whose thumbprint equals the keyid in Signature-Input.
  4. Rebuild the signature base from the request's method, host, path and Signature-Agent header, and check the Ed25519 signature.
  5. Reject it if expires has passed (signatures are valid for about 60 seconds) or the tag is not web-bot-auth.

CDNs and bot-management products that support Web Bot Auth do these steps for you. Keys are rotated with an overlap: a new key appears in the directory at least a day before it signs anything, and an old key stays listed for a day after its last signature. A key that may have been exposed is replaced at once and removed from the directory; there is no revocation, so a copy of the directory you cached still trusts it until you refresh.

7. Removal requests and abuse reports

Blocking InfusBot in robots.txt is the fastest way to stop new crawling. If you also want content from your site removed from our customers' knowledge bases, or you believe InfusBot is misbehaving, contact us. Please include:

  • the domain, and example URLs if they help us understand the request (a block always covers the whole domain)
  • what you are asking for: stop crawling, or stop crawling and delete content that has already been indexed
  • how you are connected to the site (for example, its owner)
  • for misbehavior: approximate times, the URLs requested, and the user agent you saw

On a valid request we block the domain for all customers, stop every source that crawls it and, if you ask, delete the pages already indexed from it.

Use the form below to prepare a request: it checks the details and then hands the request to your email app, addressed to info@infus.ai. You can also write to that address directly.

Subdomains of this domain are included.

One per line, at most 50. Leave empty for the whole domain.

What are you asking for?

How you are connected to this site.