Skip to main content
Version: 0.7.3

Hallucination Detection

A retrieval-augmented (RAG) answer can say things that none of the retrieved chunks support. The SDK has what you need to catch that while the answer streams: per-claim grounding scores, a final requires_review flag, and an optional hook for a semantic verifier. This recipe combines them. For each answer it decides whether to show it, flag it for a human, or withhold it.

The examples on this page are tested. The SDK's unit tests run them against a stubbed stream, and its live end-to-end tests run them against a real Engine.

How it works​

  1. One message per retrieved chunk. Every chunk goes into its own message (Context [n]: …). Each attribution source then points at exactly one chunk, and source_id n is chunk n, because message 0 is the system prompt.

  2. Stream with evidence. streamWithEvidence / stream_with_evidence yields the same text chunks as a plain stream, so you render the answer while it arrives. It also yields a claim_evidence event for each finished sentence and a response_evidence report at the end.

  3. Gate on the evidence. Each claim gets an action. The answer gets the strictest of them, and is never shown unflagged when the report sets requires_review:

    ClaimAction
    verdict contradicted (from a verifier)withhold
    verdict supported (from a verifier), or grounding ≥ SHOW_AT (0.65)show
    grounding < WITHHOLD_BELOW (0.3)withhold
    anything in betweenflag
    the answer contains no claimwithhold
  4. Optional: verify semantically. A verifier, for example an entailment model or an LLM judge, checks each claim against the chunks that attribution linked it to. It can raise a claim to supported or overrule it as contradicted.

The complete example​

Set INVERA_BASE_URL (and INVERA_API_KEY if your Engine deployment needs a key), then run the file: npx tsx hallucination-detection.ts or python hallucination_detection.py.

hallucination-detection.ts
// Hallucination detection: answer from retrieved chunks and gate the answer on its evidence.
import {
InveraClient,
type ClaimEvidence,
type ClaimVerifier,
type CompletionRequest,
type EvidenceOptions,
type ResponseEvidence,
type SemanticVerification,
} from "@layerwise-ai/invera-sdk";
import { pathToFileURL } from "node:url";

export const SYSTEM_PROMPT =
"Answer the question in one sentence using only the context passages. " +
"If the passages do not contain the answer, say that you do not know.";

// Starting points only: calibrate both on your own labelled questions.
export const SHOW_AT = 0.65; // every claim at least this grounded: show the answer
export const WITHHOLD_BELOW = 0.3; // any claim less grounded than this: withhold the answer

export type Action = "show" | "flag" | "withhold";
export const SEVERITY: Record<Action, number> = { show: 0, flag: 1, withhold: 2 };

/** The streamed answer, what to do with it, and the evidence behind that decision. */
export interface GatedAnswer {
action: Action;
text: string;
claims: ClaimEvidence[];
report: ResponseEvidence;
}

/** Put every retrieved chunk in its own message, so a source id names one chunk. */
export function buildRequest(
question: string,
chunks: readonly string[],
systemPrompt: string = SYSTEM_PROMPT,
): CompletionRequest {
return {
messages: [
{ role: "system", content: systemPrompt },
...chunks.map((chunk, index) => ({
role: "user" as const,
content: `Context [${index + 1}]:\n${chunk}`,
})),
{ role: "user", content: question },
],
maxTokens: 128,
topKAttributions: 16,
};
}

/** Return the 1-based numbers of the retrieved chunks a claim draws on, strongest first. */
export function citedChunks(claim: ClaimEvidence, chunkCount: number): number[] {
return claim.sources
.filter((source) => source.sourceType === "message")
.map((source) => source.sourceId)
.filter((id): id is number => id !== null && id >= 1 && id <= chunkCount);
}

/** Decide per claim: a verifier verdict wins, otherwise the grounding score decides. */
export function claimAction(claim: ClaimEvidence): Action {
if (claim.verdict === "contradicted") return "withhold";
if (claim.verdict === "supported" || claim.groundingScore >= SHOW_AT) return "show";
if (claim.groundingScore < WITHHOLD_BELOW) return "withhold";
return "flag";
}

/** Stream an answer with evidence and decide whether to show, flag, or withhold it. */
export async function answerWithEvidence(
client: Pick<InveraClient, "chat">,
question: string,
chunks: readonly string[],
options?: EvidenceOptions,
systemPrompt: string = SYSTEM_PROMPT,
): Promise<GatedAnswer> {
let text = "";
let report: ResponseEvidence | undefined;
const events = client.chat.completions.streamWithEvidence(buildRequest(question, chunks, systemPrompt), options);
for await (const event of events) {
if (event.type === "completion_chunk" && event.chunk.type === "text") {
text += event.chunk.content ?? ""; // render the streaming text here
} else if (event.type === "response_evidence") {
report = event.evidence;
}
}
if (!report) throw new Error("The evidence stream ended without a response_evidence event.");

// An answer without claims has nothing that grounds it.
const actions = report.claims.length > 0 ? report.claims.map(claimAction) : (["withhold"] as Action[]);
let action = actions.reduce((worst, next) => (SEVERITY[next] > SEVERITY[worst] ? next : worst));
if (report.requiresReview && action === "show") action = "flag";
return { action, text, claims: report.claims, report };
}

/** Scores a (premise, hypothesis) pair: probabilities for entailment, contradiction, neutral. */
export type EntailmentModel = (
premise: string,
hypothesis: string,
) => Promise<Record<string, number>> | Record<string, number>;

/** Judge each claim against the context sources attribution linked it to. */
export function makeVerifier(nli: EntailmentModel): ClaimVerifier {
return async ({ claim, sources }): Promise<SemanticVerification> => {
const premise = sources
.filter((source) => source.sourceType === "message" && source.role === "user")
.map((source) => source.text)
.join("\n");
if (!premise) return { relation: "unknown", score: 0 };
const scores = await nli(premise, claim);
const [label, score] = Object.entries(scores).reduce((best, next) => (next[1] > best[1] ? next : best));
const relation = label === "entailment" ? "entails" : label === "contradiction" ? "contradicts" : "unknown";
return { relation, score };
};
}

export async function main(): Promise<void> {
const client = new InveraClient({
baseUrl: process.env.INVERA_BASE_URL!,
// Only when the Engine deployment requires an API key.
apiKey: process.env.INVERA_API_KEY,
});
const chunks = [ // in your application: the top results of your retriever
"The Vellmar ferry line opened in 1931 and connects Ostrand with the island of Kessel.",
"The Kessel lighthouse was designed by the engineer Mira Talbot and first lit in 1874.",
"Kessel island has about 2,300 residents, most of whom work in fishing.",
];
const answer = await answerWithEvidence(client, "Who designed the Kessel lighthouse?", chunks);
console.log(`[${answer.action}] ${answer.text}`);
for (const claim of answer.claims) {
console.log(
` ${claim.verdict.padEnd(18)} ${claim.groundingScore.toFixed(2)} ` +
`chunks=${JSON.stringify(citedChunks(claim, chunks.length))} ${JSON.stringify(claim.text)}`,
);
}
}

// Run with Node.js (for example `npx tsx hallucination-detection.ts`).
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) await main();

For the Kessel lighthouse question, the answer repeats the second chunk. Each claim scores above 0.65 and cites chunk 2, so the action is show. If you remove that chunk, the model answers that it does not know. That sentence follows the prompt's instructions, is grounded in the system prompt, and is shown, which is the outcome you want. An answer that invents a name instead has no context to draw on: its claims score low and the answer is withheld. Words the answer only repeats from the question do not count as grounding, so echoing the question does not make an invented answer look grounded.

Acting on the result​

  • show: render the answer as it streamed. To show where a claim comes from, highlight its output_span and link the chunks from citedChunks / cited_chunks.
  • flag: render the answer with a warning and highlight the claims whose action is flag. If your product has a review queue, send the answer there.
  • withhold: replace the answer with a fallback, for example "I could not find this in the documents", and log the claims for analysis.

The action for the whole answer is only known once the stream has ended. If you render while streaming, make sure you can still take the answer back. Or hold back each sentence until its claim_evidence event arrives, and stop at the first claim whose action is withhold.

Optional: add a semantic verifier​

makeVerifier / make_verifier in the example turns any entailment scorer into a verifier. The scorer takes the cited chunks as the premise and the claim as the hypothesis, and returns probabilities for entailment, contradiction, and neutral. The label with the highest probability decides. A relation of entails or contradicts with a score of at least 0.5 takes precedence over the grounding score.

The two snippets below only show how to wire in a scorer. Unlike the complete example, they are not tested. In Python, an NLI cross-encoder from Hugging Face works directly (it needs PyTorch):

from transformers import pipeline

from invera_sdk import EvidenceOptions

classify = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-base", top_k=None)


def nli(premise: str, hypothesis: str) -> dict[str, float]:
scores = classify({"text": premise, "text_pair": hypothesis})
return {item["label"].lower(): item["score"] for item in scores}


answer = answer_with_evidence(client, question, chunks, EvidenceOptions(verifier=make_verifier(nli)))

In TypeScript, the scorer may be async. You can call a hosted NLI model there, or an LLM judge that you ask for the same three probabilities:

const answer = await answerWithEvidence(client, question, chunks, {
verifier: makeVerifier(async (premise, hypothesis) => judge(premise, hypothesis)),
});

The verifier runs once for each claim, inside the stream. A slow verifier therefore delays the later claim events, but not the text chunks that come before them.

To call a judge only for the claims attribution does not settle, instead of for every claim, see Escalate unsupported claims to an LLM judge.

Choose thresholds on your own data​

SHOW_AT = 0.65 and WITHHOLD_BELOW = 0.3 are the SDK's verdict boundaries. Use them as a starting point, not as calibrated values:

  1. Collect 100–200 real questions together with the chunks your retriever returns for them. Include questions the chunks cannot answer.
  2. Run the recipe and record each claim's grounding_score. Label each claim by hand: does the cited chunk support it?
  3. Choose SHOW_AT for the precision you need, for example "95% of shown claims are supported". Then choose WITHHOLD_BELOW so that most of the unsupported claims fall below it. Everything in between is flagged. Check that your reviewers can handle that share.
  4. Recalibrate when you change the model, the prompt, the chunking, or top_k_attributions. All four shift the scores.

What the signal does not cover​

Grounding measures provenance: how strongly the generated words come from the context. It does not measure truth (see Evidence and verdicts). Keep these limits in mind:

  • Grounded in a wrong source. If the retriever returns an outdated or wrong chunk, an answer that copies it is fully grounded. Attribution cannot tell, and a verifier cannot either, because it checks against the same chunk. Retrieval quality and source freshness remain your responsibility.
  • Right words, wrong combination. A claim can join words from two chunks into a statement neither chunk makes. For example, one chunk gives the battery capacity of the Orvex T5 and another that of the Orvex T7; the answer pairs the T5 with the T7's capacity. Every word is grounded, so the score stays high. A semantic verifier catches this kind of error.
  • Correct, but from model knowledge. A true statement that the model knows from training has no source in the context. It scores low, so it is flagged or withheld. A correct claim that mostly repeats the question can score low as well, for example "Yes, that is correct." The LLM judge escalation recipe resolves such flags.
  • Grounded in the instructions. An "I do not know" that follows the system prompt is grounded. Grounding says the answer came from the conversation, not that it answers the question. If you need to tell refusals apart, for example to log retrieval gaps, detect them separately.
  • Omissions. The evidence judges what the answer says, not what it leaves out. An incomplete answer can be fully grounded.
  • Sentence granularity. A claim is one sentence. If a sentence mixes a grounded part with an invented detail, its score averages over both. Short answers (as in the example prompt) keep claims precise.

How the attribution is tested​

The live end-to-end suites of both SDKs run three RAG questions with distractor chunks against the configured Engine. For each word of the answer (a name, a year, a capacity), they check that the matching word of the supporting chunk is among the top input attributions of that output word. For the claim as a whole, they check that it is among the claim's top-ranked input words. The questions and the thresholds per model are kept in sdk/test-fixtures/attribution-ranking.json.