Hallucination Detection
A retrieval-augmented (RAG) answer can say things that none of the retrieved
chunks support. The SDK has what you need to catch that while the answer
streams: per-claim grounding scores, a final requires_review flag, and an
optional hook for a semantic verifier. This recipe combines them. For each
answer it decides whether to show it, flag it for a human, or
withhold it.
The examples on this page are tested. The SDK's unit tests run them against a stubbed stream, and its live end-to-end tests run them against a real Engine.
How it works
-
One message per retrieved chunk. Every chunk goes into its own message (
Context [n]: …). Each attribution source then points at exactly one chunk, andsource_idn is chunk n, because message 0 is the system prompt. -
Stream with evidence.
streamWithEvidence/stream_with_evidenceyields the same text chunks as a plain stream, so you render the answer while it arrives. It also yields aclaim_evidenceevent for each finished sentence and aresponse_evidencereport at the end. -
Gate on the evidence. Each claim gets an action. The answer gets the strictest of them, and is never shown unflagged when the report sets
requires_review:Claim Action verdict contradicted(from a verifier)withhold verdict supported(from a verifier), or grounding ≥SHOW_AT(0.65)show grounding < WITHHOLD_BELOW(0.3)withhold anything in between flag the answer contains no claim withhold -
Optional: verify semantically. A verifier, for example an entailment model or an LLM judge, checks each claim against the chunks that attribution linked it to. It can raise a claim to
supportedor overrule it ascontradicted.
The complete example
Set INVERA_BASE_URL (and INVERA_API_KEY if your Engine deployment needs a
key), then run the file: npx tsx hallucination-detection.ts or
python hallucination_detection.py.
- TypeScript
- Python
// Hallucination detection: answer from retrieved chunks and gate the answer on its evidence.
import {
InveraClient,
type ClaimEvidence,
type ClaimVerifier,
type CompletionRequest,
type EvidenceOptions,
type ResponseEvidence,
type SemanticVerification,
} from "@layerwise-ai/invera-sdk";
import { pathToFileURL } from "node:url";
export const SYSTEM_PROMPT =
"Answer the question in one sentence using only the context passages. " +
"If the passages do not contain the answer, say that you do not know.";
// Starting points only: calibrate both on your own labelled questions.
export const SHOW_AT = 0.65; // every claim at least this grounded: show the answer
export const WITHHOLD_BELOW = 0.3; // any claim less grounded than this: withhold the answer
export type Action = "show" | "flag" | "withhold";
export const SEVERITY: Record<Action, number> = { show: 0, flag: 1, withhold: 2 };
/** The streamed answer, what to do with it, and the evidence behind that decision. */
export interface GatedAnswer {
action: Action;
text: string;
claims: ClaimEvidence[];
report: ResponseEvidence;
}
/** Put every retrieved chunk in its own message, so a source id names one chunk. */
export function buildRequest(
question: string,
chunks: readonly string[],
systemPrompt: string = SYSTEM_PROMPT,
): CompletionRequest {
return {
messages: [
{ role: "system", content: systemPrompt },
...chunks.map((chunk, index) => ({
role: "user" as const,
content: `Context [${index + 1}]:\n${chunk}`,
})),
{ role: "user", content: question },
],
maxTokens: 128,
topKAttributions: 16,
};
}
/** Return the 1-based numbers of the retrieved chunks a claim draws on, strongest first. */
export function citedChunks(claim: ClaimEvidence, chunkCount: number): number[] {
return claim.sources
.filter((source) => source.sourceType === "message")
.map((source) => source.sourceId)
.filter((id): id is number => id !== null && id >= 1 && id <= chunkCount);
}
/** Decide per claim: a verifier verdict wins, otherwise the grounding score decides. */
export function claimAction(claim: ClaimEvidence): Action {
if (claim.verdict === "contradicted") return "withhold";
if (claim.verdict === "supported" || claim.groundingScore >= SHOW_AT) return "show";
if (claim.groundingScore < WITHHOLD_BELOW) return "withhold";
return "flag";
}
/** Stream an answer with evidence and decide whether to show, flag, or withhold it. */
export async function answerWithEvidence(
client: Pick<InveraClient, "chat">,
question: string,
chunks: readonly string[],
options?: EvidenceOptions,
systemPrompt: string = SYSTEM_PROMPT,
): Promise<GatedAnswer> {
let text = "";
let report: ResponseEvidence | undefined;
const events = client.chat.completions.streamWithEvidence(buildRequest(question, chunks, systemPrompt), options);
for await (const event of events) {
if (event.type === "completion_chunk" && event.chunk.type === "text") {
text += event.chunk.content ?? ""; // render the streaming text here
} else if (event.type === "response_evidence") {
report = event.evidence;
}
}
if (!report) throw new Error("The evidence stream ended without a response_evidence event.");
// An answer without claims has nothing that grounds it.
const actions = report.claims.length > 0 ? report.claims.map(claimAction) : (["withhold"] as Action[]);
let action = actions.reduce((worst, next) => (SEVERITY[next] > SEVERITY[worst] ? next : worst));
if (report.requiresReview && action === "show") action = "flag";
return { action, text, claims: report.claims, report };
}
/** Scores a (premise, hypothesis) pair: probabilities for entailment, contradiction, neutral. */
export type EntailmentModel = (
premise: string,
hypothesis: string,
) => Promise<Record<string, number>> | Record<string, number>;
/** Judge each claim against the context sources attribution linked it to. */
export function makeVerifier(nli: EntailmentModel): ClaimVerifier {
return async ({ claim, sources }): Promise<SemanticVerification> => {
const premise = sources
.filter((source) => source.sourceType === "message" && source.role === "user")
.map((source) => source.text)
.join("\n");
if (!premise) return { relation: "unknown", score: 0 };
const scores = await nli(premise, claim);
const [label, score] = Object.entries(scores).reduce((best, next) => (next[1] > best[1] ? next : best));
const relation = label === "entailment" ? "entails" : label === "contradiction" ? "contradicts" : "unknown";
return { relation, score };
};
}
export async function main(): Promise<void> {
const client = new InveraClient({
baseUrl: process.env.INVERA_BASE_URL!,
// Only when the Engine deployment requires an API key.
apiKey: process.env.INVERA_API_KEY,
});
const chunks = [ // in your application: the top results of your retriever
"The Vellmar ferry line opened in 1931 and connects Ostrand with the island of Kessel.",
"The Kessel lighthouse was designed by the engineer Mira Talbot and first lit in 1874.",
"Kessel island has about 2,300 residents, most of whom work in fishing.",
];
const answer = await answerWithEvidence(client, "Who designed the Kessel lighthouse?", chunks);
console.log(`[${answer.action}] ${answer.text}`);
for (const claim of answer.claims) {
console.log(
` ${claim.verdict.padEnd(18)} ${claim.groundingScore.toFixed(2)} ` +
`chunks=${JSON.stringify(citedChunks(claim, chunks.length))} ${JSON.stringify(claim.text)}`,
);
}
}
// Run with Node.js (for example `npx tsx hallucination-detection.ts`).
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) await main();
"""Hallucination detection: answer from retrieved chunks and gate the answer on its evidence."""
from __future__ import annotations
import os
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import Literal
from invera_sdk import (
ChatMessage,
ClaimEvidence,
ClaimVerificationInput,
CompletionRequest,
EvidenceOptions,
InveraClient,
ResponseEvidence,
SemanticVerification,
)
SYSTEM_PROMPT = (
"Answer the question in one sentence using only the context passages. "
"If the passages do not contain the answer, say that you do not know."
)
# Starting points only: calibrate both on your own labelled questions.
SHOW_AT = 0.65 # every claim at least this grounded: show the answer
WITHHOLD_BELOW = 0.3 # any claim less grounded than this: withhold the answer
Action = Literal["show", "flag", "withhold"]
SEVERITY: dict[Action, int] = {"show": 0, "flag": 1, "withhold": 2}
@dataclass(frozen=True)
class GatedAnswer:
"""The streamed answer, what to do with it, and the evidence behind that decision."""
action: Action
text: str
claims: tuple[ClaimEvidence, ...]
report: ResponseEvidence
def build_request(
question: str, chunks: Sequence[str], system_prompt: str = SYSTEM_PROMPT
) -> CompletionRequest:
"""Put every retrieved chunk in its own message, so a source id names one chunk."""
messages = [ChatMessage("system", system_prompt)]
messages += [
ChatMessage("user", f"Context [{number}]:\n{chunk}")
for number, chunk in enumerate(chunks, start=1)
]
messages.append(ChatMessage("user", question))
return CompletionRequest(messages, max_tokens=128, top_k_attributions=16)
def cited_chunks(claim: ClaimEvidence, chunk_count: int) -> list[int]:
"""Return the 1-based numbers of the retrieved chunks a claim draws on, strongest first."""
return [
source.source_id
for source in claim.sources
if source.source_type == "message"
and source.source_id is not None
and 1 <= source.source_id <= chunk_count
]
def claim_action(claim: ClaimEvidence) -> Action:
"""Decide per claim: a verifier verdict wins, otherwise the grounding score decides."""
if claim.verdict == "contradicted":
return "withhold"
if claim.verdict == "supported" or claim.grounding_score >= SHOW_AT:
return "show"
if claim.grounding_score < WITHHOLD_BELOW:
return "withhold"
return "flag"
def answer_with_evidence(
client: InveraClient,
question: str,
chunks: Sequence[str],
options: EvidenceOptions | None = None,
*,
system_prompt: str = SYSTEM_PROMPT,
) -> GatedAnswer:
"""Stream an answer with evidence and decide whether to show, flag, or withhold it."""
text = ""
report: ResponseEvidence | None = None
events = client.chat.completions.stream_with_evidence(
build_request(question, chunks, system_prompt), options
)
for event in events:
if event.type == "completion_chunk" and event.chunk.type == "text":
text += event.chunk.content or "" # render the streaming text here
elif event.type == "response_evidence":
report = event.evidence
assert isinstance(report, ResponseEvidence)
# An answer without claims has nothing that grounds it.
actions = [claim_action(claim) for claim in report.claims] or ["withhold"]
action = max(actions, key=SEVERITY.__getitem__)
if report.requires_review and action == "show":
action = "flag"
return GatedAnswer(action, text, report.claims, report)
EntailmentModel = Callable[[str, str], dict[str, float]]
"""Scores a (premise, hypothesis) pair: probabilities for entailment, contradiction, neutral."""
def make_verifier(nli: EntailmentModel) -> Callable[[ClaimVerificationInput], SemanticVerification]:
"""Judge each claim against the context sources attribution linked it to."""
def verify(item: ClaimVerificationInput) -> SemanticVerification:
premise = "\n".join(
source.text
for source in item.sources
if source.source_type == "message" and source.role == "user"
)
if not premise:
return SemanticVerification("unknown", 0.0)
scores = nli(premise, item.claim)
label = max(scores, key=scores.__getitem__)
relation = {"entailment": "entails", "contradiction": "contradicts"}.get(label, "unknown")
return SemanticVerification(relation, scores[label])
return verify
def main() -> None:
client = InveraClient(
os.environ["INVERA_BASE_URL"],
# Only when the Engine deployment requires an API key.
api_key=os.environ.get("INVERA_API_KEY"),
)
chunks = [ # in your application: the top results of your retriever
"The Vellmar ferry line opened in 1931 and connects Ostrand with the island of Kessel.",
"The Kessel lighthouse was designed by the engineer Mira Talbot and first lit in 1874.",
"Kessel island has about 2,300 residents, most of whom work in fishing.",
]
answer = answer_with_evidence(client, "Who designed the Kessel lighthouse?", chunks)
print(f"[{answer.action}] {answer.text}")
for claim in answer.claims:
print(
f" {claim.verdict:<18} {claim.grounding_score:.2f} "
f"chunks={cited_chunks(claim, len(chunks))} {claim.text!r}"
)
if __name__ == "__main__":
main()
For the Kessel lighthouse question, the answer repeats the second chunk. Each
claim scores above 0.65 and cites chunk 2, so the action is show. If you
remove that chunk, the model answers that it does not know. That sentence
follows the prompt's instructions, is grounded, and is shown, which is the
outcome you want. An answer that invents a name instead has no context to draw
on: its claims score low and the answer is withheld.
Acting on the result
- show: render the answer as it streamed. To show where a claim comes
from, highlight its
output_spanand link the chunks fromcitedChunks/cited_chunks. - flag: render the answer with a warning and highlight the claims whose
action is
flag. If your product has a review queue, send the answer there. - withhold: replace the answer with a fallback, for example "I could not find this in the documents", and log the claims for analysis.
The action for the whole answer is only known once the stream has ended. If
you render while streaming, make sure you can still take the answer back. Or
hold back each sentence until its claim_evidence event arrives, and stop at
the first claim whose action is withhold.
Optional: add a semantic verifier
makeVerifier / make_verifier in the example turns any entailment scorer
into a verifier. The scorer takes the cited chunks as the premise and the claim
as the hypothesis, and returns probabilities for entailment,
contradiction, and neutral. The label with the highest probability
decides. A relation of entails or contradicts with a score of at least 0.5
takes precedence over the grounding score.
The two snippets below only show how to wire in a scorer. Unlike the complete example, they are not tested. In Python, an NLI cross-encoder from Hugging Face works directly (it needs PyTorch):
from transformers import pipeline
from invera_sdk import EvidenceOptions
classify = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-base", top_k=None)
def nli(premise: str, hypothesis: str) -> dict[str, float]:
scores = classify({"text": premise, "text_pair": hypothesis})
return {item["label"].lower(): item["score"] for item in scores}
answer = answer_with_evidence(client, question, chunks, EvidenceOptions(verifier=make_verifier(nli)))
In TypeScript, the scorer may be async. You can call a hosted NLI model there, or an LLM judge that you ask for the same three probabilities:
const answer = await answerWithEvidence(client, question, chunks, {
verifier: makeVerifier(async (premise, hypothesis) => judge(premise, hypothesis)),
});
The verifier runs once for each claim, inside the stream. A slow verifier therefore delays the later claim events, but not the text chunks that come before them.
To call a judge only for the claims attribution does not settle, instead of for every claim, see Escalate unsupported claims to an LLM judge.
Choose thresholds on your own data
SHOW_AT = 0.65 and WITHHOLD_BELOW = 0.3 are the SDK's verdict boundaries.
Use them as a starting point, not as calibrated values:
- Collect 100–200 real questions together with the chunks your retriever returns for them. Include questions the chunks cannot answer.
- Run the recipe and record each claim's
grounding_score. Label each claim by hand: does the cited chunk support it? - Choose
SHOW_ATfor the precision you need, for example "95% of shown claims are supported". Then chooseWITHHOLD_BELOWso that most of the unsupported claims fall below it. Everything in between is flagged. Check that your reviewers can handle that share. - Recalibrate when you change the model, the prompt, the chunking, or
top_k_attributions. All four shift the scores.
What the signal does not cover
Grounding measures provenance: how strongly the generated words come from the context. It does not measure truth (see Evidence and verdicts). Keep these limits in mind:
- Grounded in a wrong source. If the retriever returns an outdated or wrong chunk, an answer that copies it is fully grounded. Attribution cannot tell, and a verifier cannot either, because it checks against the same chunk. Retrieval quality and source freshness remain your responsibility.
- Right words, wrong combination. A claim can join words from two chunks into a statement neither chunk makes, for example the T5's model name with the T7's battery capacity. Every word is grounded, so the score stays high. A semantic verifier catches this kind of error.
- Correct, but from model knowledge. A true statement that the model knows from training has no source in the context. It scores low and is withheld, as the prompt intends.
- Grounded in the question or the instructions. A claim that only repeats the question, or an "I do not know" that follows the system prompt, is grounded as well. Grounding says the answer came from the conversation, not that it answers the question. If you need to tell refusals apart, for example to log retrieval gaps, detect them separately. The same applies to a claim from model knowledge that repeats the words of the question: it can score high. The LLM judge escalation recipe catches such claims by their low share of attribution from the chunks.
- Omissions. The evidence judges what the answer says, not what it leaves out. An incomplete answer can be fully grounded.
- Sentence granularity. A claim is one sentence. If a sentence mixes a grounded part with an invented detail, its score averages over both. Short answers (as in the example prompt) keep claims precise.
How the attribution is tested
The live end-to-end suites of both SDKs run three RAG questions with
distractor chunks against the configured Engine. For each word of the answer
(a name, a year, a capacity), they check that the matching word of the
supporting chunk is among the top input attributions of that output word. For
the claim as a whole, they check that it is among the claim's top-ranked input
words. The questions and the thresholds per model are kept in
sdk/test-fixtures/attribution-ranking.json.