Evidence and Verdicts
The evidence layer reconstructs semantic evidence from attributed completion chunks — entirely client-side. This page explains what the scores are made of, so you can decide what to do with them.
Honest by design
Attribution grounding measures provenance, not truth: it tells you how
strongly generated words trace back to conversation context. A claim can be
perfectly grounded in a source that is itself wrong, and a correct claim can
be ungrounded because the model knew it from training. That is why verdicts
without a verifier never say "supported" — only grounded,
partially_grounded, or ungrounded.
How claims are segmented
Visible text is accumulated until a claim becomes stable: at a newline, at
sentence-ending punctuation (., !, ? followed by whitespace), or at the
max_claim_chars limit (default 500, split at the last space). Each claim
records its exact character span in the assistant text (output_span), so
you can highlight it in your UI.
The grounding score
Each claim's grounding_score in [0, 1] combines four signals:
grounding = (0.65 · coverage + 0.25 · source_share + 0.10 · context_overlap)
· (0.75 + 0.25 · source_quality)
attribution_coverage— the term-weighted share of the claim that has any positive input attribution at all. The dominant signal: unattributed stretches of text drag it down.- source share — input-attribution mass relative to total mass. Text that mostly draws on the model's own earlier output scores low.
context_overlap— the share of the claim's important terms (numbers, negations, capitalized and long words) that literally appear in the attributed source texts. A weak lexical cross-check on top of attribution.- source quality — an attribution-weighted average over the claim's sources: tool results 1.0, system messages 0.9, user messages 0.75, assistant messages 0.5, tool descriptions 0.25. Quality never raises the score: a claim drawn only from tool results keeps its full score, and lower-quality sources reduce it by at most 25%.
The question itself is not a source: a claim that only repeats the question gains no grounding from it (see What counts as context).
Two further fields do not enter the score but are useful signals:
instruction_grounding (how much of the input mass comes from system and
user messages) and sources (the ranked source list with the exact
attributed words).
Verdicts
| Verdict | Produced when |
|---|---|
contradicted | A verifier reported contradicts with score ≥ 0.5. |
supported | A verifier reported entails with score ≥ 0.5. |
grounded | Grounding score ≥ 0.65. |
partially_grounded | Grounding score ≥ 0.3. |
ungrounded | Grounding score below 0.3. |
Verifier verdicts take precedence over attribution thresholds; claims judged
with a verifier report method: "attribution_and_verifier". The unknown
verdict value is reserved for host code that post-processes evidence — the
SDK itself never emits it.
Output-kind detection
The output_kind event fires once, as soon as the kind of the first
output is observed — tool_call on the first tool-call chunk, text on the
first non-whitespace text. Its definitive flag is true because the kind was
directly observed rather than guessed. A response may still continue with
the other kind afterwards; the final ResponseEvidence.output_kind reports
text, tool_call, or mixed over the whole response.
Tool-call evidence
Completed tool calls are assessed on three independent axes:
- Structure — JSON parsing and schema validation against your tool
definitions (types, enums, required arguments, additional properties),
reported as machine-readable
schema_issues, pluscompleteness(the share of required arguments present; 0 for unparseable or unknown calls). - Grounding —
parameterslinks each argument leaf to its context sources (share of input mass plus a lexical match bonus), andintent_groundingmeasures how much the call as a whole traces back to system, user, and tool messages. - Policy — your local
ToolPolicy(risk level, approval requirement, side effect, sensitive parameters) producesrisk_reasonsand drives the finaldecision:blockon schema violations,reviewwhen policy demands approval, otherwiseallow.
The final report
The closing response_evidence event aggregates everything:
overall_grounding averages the claim grounding scores (or the tool-call
intent groundings when the response contains no claims), and
requires_review is set when any claim is contradicted, ungrounded, or
unknown, or any tool call decided other than allow.
What counts as context
By default, claim evidence ignores attribution to the question, the last user
message of the request. The question is left out of the grounding score, of
sources, and of instruction_grounding. System messages, earlier user
messages, and tool results all count. This fits the common RAG layouts, where
the retrieved text sits in the system message, in tool results, or in user
messages before the question.
If the last user message is at least as long as all other system, user, and tool messages together, it carries the retrieved text itself (documents and question in one message). It then counts like any other message, and the scores are the same as in SDK 0.7.2. Tool-call evidence always counts the question, because a tool call acts on what the user asked.
To count every message, as SDK 0.7.2 and earlier did, set the context
option to "conversation": EvidenceOptions(context="conversation") in
Python, { context: "conversation" } in TypeScript. The default is "auto".