Skip to main content

Chat with Citations

Grounded chat uses the endpoint you already use, POST /v1/chat/completions, with two optional fields. There is no separate RAG chat endpoint, and a request without knowledge_base_ids is an ordinary chat request that never touches your knowledge bases.

Request fields​

{
"model": "your-chat-model",
"messages": [
{ "role": "system", "content": "You are the HR assistant of Example Corp." },
{ "role": "user", "content": "How many days of annual leave do new employees get?" }
],
"knowledge_base_ids": ["3f6c2b1e-8d4a-4c1f-9e2b-7a5d6c3b2a10"],
"retrieval": {
"top_k": 5,
"filter": { "must": [{ "key": "department", "eq": "hr" }] },
"document_ids": null,
"max_source_tokens": 3000
},
"stream": false
}
FieldDefaultRule
knowledge_base_ids—1–5 unique knowledge-base ids. Absent or [] = ordinary chat.
retrieval.top_k51–20 candidate chunks, merged across all listed knowledge bases.
retrieval.filternoneA metadata filter.
retrieval.document_idsnoneRestrict retrieval to up to 100 documents of the listed knowledge bases.
retrieval.max_source_tokens3,000256–8,000. Token budget for source text in the prompt; sources are added in score order until the next one would exceed it.

Both fields are removed before the request reaches the model, so the model provider never sees them.

What the platform does​

  1. Checks the key: it needs inference:chat and knowledge:read (or knowledge:write), and every listed knowledge base must be on its access list. One refused knowledge base refuses the whole request.
  2. Uses only the last user message as the retrieval query (its first 2,000 characters; the full message still goes to the model). Earlier turns are passed to the model unchanged but do not influence retrieval. The last message must be a user message with text.
  3. Retrieves the best chunks across the listed knowledge bases — this is one query, however many knowledge bases you list.
  4. Places the sources that fit max_source_tokens in the prompt, numbered [1], [2], …, and adds a fixed instruction telling the model to answer from them, cite them by number, say so when the sources do not contain the answer, and answer in the language of the question.
  5. Calls the model and returns the answer with citations — only the sources actually placed in the prompt, numbered as in the prompt.

If retrieval cannot run (knowledge bases temporarily unreachable, search index or embedding failure), the request fails with 503 — the platform never silently answers without sources.

Citations​

A citation object:

{
"index": 1,
"knowledge_base_id": "3f6c2b1e-8d4a-4c1f-9e2b-7a5d6c3b2a10",
"doc_id": "1d2e3f40-5a6b-4c7d-8e9f-0a1b2c3d4e5f",
"document_name": "handbook.pdf",
"page": 12,
"chunk_index": 41,
"text": "Employees with less than five years of service receive 14 working days …",
"score": 0.0312
}

index is the number the model uses in its answer ([1], [2][3]). page is null for formats without pages. Use doc_id with GET /v1/knowledge_bases/{kb_id}/documents/{doc_id}/content to offer the original file.

A citation may also carry section (string or null, as in search hits) and location — the chunk's position in its source, with the fields that fit the file type (page, slide, slide_title, sheet, row_start, row_end, line_start, line_end, heading_path; see Chunks and locations). Both are optional: treat a missing field as unknown and ignore location keys you do not recognise. A citation from a spreadsheet:

{
"index": 2,
"knowledge_base_id": "3f6c2b1e-8d4a-4c1f-9e2b-7a5d6c3b2a10",
"doc_id": "7a8b9c0d-1e2f-4a3b-8c4d-5e6f7a8b9c0d",
"document_name": "price-list.xlsx",
"page": null,
"chunk_index": 3,
"section": "Devices",
"text": "…",
"score": 0.0287,
"location": { "sheet": "Devices", "row_start": 4, "row_end": 13 }
}

A citation of a page from a website source also carries source: {source_id, url, fetched_at, verified}, set by the platform (never by metadata). Link the citation to source.url, and label pages with verified: false as third-party content. For uploaded documents source is absent or null.

An empty citations array means retrieval ran and found nothing relevant; the model is instructed to say that the knowledge base does not contain the answer instead of guessing.

Non-streaming​

citations is a top-level field of the chat.completion object, next to choices and usage:

{
"id": "chatcmpl-9f2c…",
"object": "chat.completion",
"created": 1790157600,
"model": "your-chat-model",
"choices": [ … ],
"usage": { … },
"citations": [ { "index": 1, … } ]
}

Streaming​

With "stream": true the citations arrive in the first server-sent event, before any content. It is an ordinary chat.completion.chunk whose only delta is the assistant role with empty content, plus a top-level citations array. There is no event: line, so OpenAI-compatible clients that ignore unknown fields keep working unchanged:

data: {"id":"chatcmpl-9f2c…","object":"chat.completion.chunk","created":1790157600,"model":"your-chat-model","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"citations":[{"index":1,"knowledge_base_id":"3f6c…","doc_id":"1d2e…","document_name":"handbook.pdf","page":12,"chunk_index":41,"text":"Employees with less than five years …","score":0.0312}]}

data: {"id":"chatcmpl-9f2c…","object":"chat.completion.chunk", … the model's chunks … }

data: [DONE]
  • The citations chunk is sent exactly once, even when citations is [].
  • Its id, created and model equal those of the following chunks.
  • Detect it by the presence of the top-level citations key.
  • An error after the stream has started uses the normal mid-stream error frame (see Streaming errors); citations already sent cannot be recalled.

Python​

from openai import OpenAI

client = OpenAI(api_key="sk-your-api-key", base_url="https://api.infus.ai/v1")

stream = client.chat.completions.create(
model="your-chat-model",
messages=[{"role": "user", "content": "How many days of annual leave do new employees get?"}],
extra_body={"knowledge_base_ids": ["3f6c2b1e-8d4a-4c1f-9e2b-7a5d6c3b2a10"]},
stream=True,
)

citations = []
for chunk in stream:
extra = chunk.model_extra or {}
if "citations" in extra:
citations = extra["citations"] # first chunk
continue
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)

print()
for c in citations:
print(f"[{c['index']}] {c['document_name']} (page {c['page']})")

JavaScript​

import OpenAI from "openai";

const client = new OpenAI({
apiKey: process.env.API_KEY,
baseURL: "https://api.infus.ai/v1",
});

const stream = await client.chat.completions.create({
model: "your-chat-model",
messages: [{ role: "user", content: "How many days of annual leave do new employees get?" }],
// @ts-expect-error knowledge_base_ids is a platform extension
knowledge_base_ids: ["3f6c2b1e-8d4a-4c1f-9e2b-7a5d6c3b2a10"],
stream: true,
});

let citations = [];
for await (const chunk of stream) {
if (chunk.citations) {
citations = chunk.citations; // first chunk
continue;
}
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
console.log("\n", citations.map((c) => `[${c.index}] ${c.document_name}`));

Restrictions​

CombinationResult
knowledge_base_ids with tools, functions or tool_choice422 tools_not_supported_with_knowledge — retrieved text must never be able to trigger tool calls.
knowledge_base_ids with a background/async job422 validation_failed (param: background) — background jobs cannot return citations.
Last message not a user message with text422 retrieval_query_missing
More than 5 knowledge bases422 too_many_knowledge_bases

Errors​

SituationStatus and code
Key lacks inference:chat or a knowledge scope403 scope_missing
Any knowledge base not on the key's access list403 kb_not_in_allowlist (whole request)
Any knowledge base missing or not yours404 not_found (whole request)
Knowledge bases switched off on the platform and your organization has none404 module_disabled (with existing knowledge bases, grounded chat keeps working read-only)
Any knowledge base deleting, suspended or frozen409 kb_deleting / kb_suspended / trial_kb_frozen
Query allowance or money refused for the retrieval402 — trial_exhausted, insufficient_credits, credit_limit_exceeded, payg_spend_cap_exceeded, payment_suspended
Invalid extension input422 — see above, plus filter_invalid, validation_failed
Knowledge query rate limit429 rate_limited
Knowledge bases temporarily unreachable503 knowledge_unavailable
Search index or embedding failure503 retrieval_unavailable
Billing gave no answer for the retrieval (unreachable, timeout or a 5xx)By default nothing: the retrieval is served and its query charge is settled once billing answers again (unless your platform operator switched this off). An answer billing did give still counts: a refusal the platform cannot map, a malformed answer or a 4xx is 503 billing_unavailable, and so is measurement_debt

Full list: Error Codes.

Billing​

A grounded answer costs one query for the retrieval plus the model's normal token price. The source text placed in the prompt counts as input tokens of the model. If retrieval succeeded and the model call then fails or is refused, the query stays charged; the model part follows the normal inference rules. There is no hidden extra fee. See Pricing and Billing.

Feedback​

Let users rate an answer with POST /v1/knowledge_feedback:

{
"request_id": "chatcmpl-9f2c…",
"rating": "up",
"knowledge_base_ids": ["3f6c2b1e-8d4a-4c1f-9e2b-7a5d6c3b2a10"],
"document_ids": ["1d2e3f40-5a6b-4c7d-8e9f-0a1b2c3d4e5f"]
}

request_id is the completion id. One vote per completion; a new vote replaces the old one. Only the ids and the rating are stored — never the question or the answer. Votes appear in the knowledge base's usage summary.

Prompt injection​

Retrieved text is treated as untrusted data, never as instructions:

  • Sources are placed inside a delimited <knowledge_sources> block in the last user message, with their text escaped so a document cannot close or open the block.
  • A fixed system instruction tells the model to treat everything inside the block as reference data and to ignore instructions, role changes, requests to reveal the prompt, or requests to call tools or visit links that appear there.
  • Tool calling is disabled for grounded requests.

This reduces, but cannot fully eliminate, the risk that a document crafted to manipulate a model changes an answer. Only upload documents from sources you trust, and treat answers as you would any model output.

Where sources go

The retrieved excerpts are sent to the chat model you select as part of its prompt. If that model is served by an external provider, the excerpts reach that provider like any other prompt content.