In one sentence
Checking that a claimed number really appears in the cited evidence does not tell you the evidence was all of the evidence — so a verifier has to track completeness as carefully as it tracks values.
Why this matters
During a live demo, our 14B model did everything right. It planned a reachability query, got a page of results back, counted the rows — carefully, correctly — and reported the count, citing the step it came from. The verifier checked the claim against the trace and approved it.
One problem. The page held 100 rows. The complete result held 343.
Nobody in that chain misbehaved. The operation returned a page because that is what it is supposed to do. The model counted what it was given. The verifier confirmed the number appeared in the cited evidence, which it did. The claim was wrong anyway, and every check passed.
The idea in plain language
The database was not hiding anything. Every paginated result it returns
carries two extra pieces of information alongside the rows: how many rows
matched in total, and whether what you are holding is the whole set.
In TGMS those are called rows_total and truncated, and
they are the honest part of this story — a partial-result marker, present
and correct in the payload the model received.
// what the operation returned — default page limit 100 {"rows": [ …100 items… ], "rows_total": 343, "truncated": true} // what the model claimed, citing that step {"type": "count", "value": 100, "evidence": ["s2"]} // ✓ the value really is in s2 …
The marker existed. What did not exist was any obligation to carry it forward. The moment those rows were counted, the count became a plain number with no memory of where it came from — and once that is lost, no downstream check can recover it.
This is not a quirk of one query. It is every SQL LIMIT, every
top-k retrieval, every sampled aggregate, every timed-out scan, every
stale replica. Databases label partial answers honestly; the labels evaporate on
the way to the conclusion.
How TGMS handles it
The mechanism is that the marker is propagated instead of dropped. When a
step reads a result that was truncated, that step is marked too, and so is every
step that depends on it — the count over those rows, the filter over that count,
all the way to the answer. In the code this downstream mark is called
truncation taint; a step records both its own truncated flag
and an upstream_truncated one inherited from its dependencies.
The claim verifier then reads that mark. It rates each claim supported, weakly supported, unsupported or unverifiable, and a claim resting on tainted evidence can be rated no higher than weakly supported, however perfect its arithmetic. The badge a reader sees changes accordingly:
What we measured
- Question
- Does propagating the partial-result marker change which claims are certified, and what else does the suite catch?
- Workload
- Generated mutations of otherwise-correct answers, one class per failure mode, plus a clean pool of 202 unmutated answers to measure false positives.
- Compared
- For the truncation class: the same cases verified twice, with completeness propagation enabled and disabled.
- Metric
- Detection — the share of injected faults refused full support. Higher is better.
- Run
- Deterministic generation and verification; case counts are what the generator could construct per class, not a chosen sample size.
What we found
The propagation carries the weight: with it enabled, every generated correct-arithmetic-over-partial-evidence case is refused full support; with it disabled, every one passes as fully supported. One class is a deliberate, documented zero.
| Failure type | Detected | Why |
|---|---|---|
| Correct count over a partial page | 15 / 15 | the marker reaches the claim and caps it |
| … the same cases, propagation disabled | 0 / 15 | nothing links the count back to a partial page |
| Value correct under the wrong belief time | 60 / 60 | claims are re-checked at the belief time the trace was pinned to |
| A real entity the evidence never mentioned | 100 / 100 | cited identifiers must appear in the cited step |
| Unit confusion — microseconds read as milliseconds | 100 / 100 | units travel with the value |
| A member dropped from a reported set | 0 / 100 | nothing in the trace is inconsistent — see below |
End to end, the effect on emitted answers is that 0 unsupported claims appear among the 199 answers this campaign emitted with the gate on, out of 282 task runs. Ungated, the same system emitted 220 answers and 21 of them carried an unsupported claim — 7.8% of the 270 task runs that produced a measurable report at all. The gate costs about one point of answer accuracy — 41.8% ungated against 40.8% gated.
What this means in practice
The transferable rule for anyone designing an API that agents will call: your service already knows when its answer is partial — make that knowledge impossible to ignore. Completeness is not the only such fact, and all of them behave the same way.
- Return the qualifier as data, never as prose. Pagination
(
rows_total,truncated), approximation and its error bound, timeouts, replica staleness and sampling rate all belong in the payload as machine-readable fields — not in a documentation note about default limits. - Make them compositional. A qualifier is only useful if it survives being computed over. Define what happens when a partial result is counted, joined or filtered — the answer is almost always “it propagates”.
- Give consumers a place to put it. If your answer format has no field for “this rests on partial data”, every consumer will drop it, and yours is the API that made the loss inevitable.
- Distinguish “correct over what I saw” from “correct over what exists”. Only one of them is what the user asked for.
What this result does not show
Under-claiming is invisible here, by construction. If a model reports four members of a set that has five, the trace is entirely self-consistent: every identifier it cites really is in the cited evidence, and nothing is truncated. Grounding checks that what was said is in the evidence, never that nothing was left out; set completeness is checkable only against the database itself. That is the 0 of 100 in the table, and it is a statement about where verification has to live rather than a bug awaiting a patch.
The zero is not a guarantee. It is 0 unsupported claims among the 199 answers emitted across 282 task runs on one task set — a measured rate on a finite sample, not a property that holds for any question. Part of it is also bought by declining to answer: gating both suppresses bad claims and reduces coverage, so the honest reading is “nothing unsupported was emitted here”, not “nothing unsupported can be emitted”.
The per-class figures come from generated mutations, cleaner than the mistakes real models make unprompted, and case counts are whatever the generator could construct.
Takeaway
A verifier that only checks values will approve confident summaries of half the truth. Completeness has to be carried — from the operation that knows the answer was partial, through every computation over it, to the badge the reader finally sees.
Evidence and reproduction
The generator and per-class tables are in tgms/eval/faults_ext.py; the propagation is in tgms/agent/executor.py and the cap in verifier.py, both pinned by regression tests in tests/test_verifier_dynamic.py — including one whose docstring is the demo described at the top of this post. The ablation flag is ablate_truncation_taint. The study is written up in the paper, and every number here is quoted from docs/site_facts.json and checked in CI.
Continue reading
Prerequisite: why AI tools need output contracts — the same argument one layer down, about what a model is allowed to read out of a result.