In one sentence: changing only the model and leaving the data, tasks and prompts alone, the system that hands the model a set of verified data operations went from answering 13.8% of tasks to 62.8%, while the best system that hands it retrieved text went from 9.6% to 27.7% — model quality converts into answer quality at very different rates depending on what the model is allowed to do with the data.
Why this matters
Connecting a language model to a store of historical data gives you two ways to spend a budget: buy a bigger model, or build a better interface between the model and the data. Teams overwhelmingly do the first, on an assumption that is rarely examined — that a stronger model will make better use of whatever you hand it. If that holds, every approach improves at roughly the same rate and the next model is always the cheapest fix. If it does not, the ranking you measured on a small model will not survive to a large one.
The idea in plain language
The questions here are not lookups. They ask which accounts messaged a given account during one week in June, or what the database believed about a relationship before a later correction arrived — so answering means composing several steps: select a window, follow relationships, group, compare against an earlier belief.
You can retrieve relevant records, paste them into the prompt as text, and let the model compose in its head. Or you can hand it a set of verified data operations — individually checked units of work such as "history of this entity" or "state as believed at this past moment" — and let it write a plan connecting them, which the system executes. TGMS does the second; a retrieval baseline does the first. The difference is where composition happens: in a context window over serialised text, or in an execution engine.
What we measured
Evidence box
- Question. As the planning model gets stronger, does answer quality improve at the same rate for an operator interface as for retrieval baselines?
- Dataset. The frozen CollegeMsg test split — 94 tasks over a 59,835-event message log, SHA-pinned before measurement, against a store rebuilt byte-identically from its event log.
- Systems. TGMS, a static-graph RAG baseline and a text-to-Cypher baseline, each driven by the same model at each size.
- Metric. Normalized typed-answer accuracy — the share of tasks whose structured answer matches the reference, with counts and values exact and interval answers credited at IoU ≥ 0.5; higher is better. One task is worth 1.1 points.
- Held constant. Store, tasks, prompts, temperature 0, operator set. Only the model changes.
- Runs. Single run, one seed per cell, no confidence interval — weaker sampling than our primary campaign, which pools three seeds with paired bootstrap intervals.
- Snapshot.
scale-sweep-collegemsg-94, Qwen2.5 family, iTiger cluster.
What we found
Across the three sizes that share a precision — 7B, 14B and 32B, all at 16-bit — TGMS answer accuracy rose from 13.8% to 34.0% to 62.8%, a factor of 4.5. The strongest baseline, text-to-Cypher, rose from 9.6% to 27.7% — nearly tripling in relative terms, but gaining about 18 points where TGMS gained 49, and never passing 27.7% at any size we ran.
scale-sweep-collegemsg-94, single run, one seed, temperature
0), answer accuracy under an operator interface rises about
4.5× from 7B to
32B at 16-bit, while the strongest retrieval baseline gains about 18 points
and stays below 28%. Boundary. The 72B point is detached and
unconnected because it is 4-bit: it differs in precision as well as size, so
it is not the continuation of this curve and the study cannot attribute its
lower score to either cause. Three points are an ordering, not a functional
form, and one seed per cell cannot resolve differences of a few
points.The correction probes — questions whose answer changes after a later correction, answerable only from what the database believed at a past moment — moved from 38% to 77% to 100%. That last figure needs its denominator said plainly: the split holds 13 probes, so 100% is 13 of 13 on one seed and a single probe is worth 7.7 points. It shows the bi-temporal machinery is usable rather than aspirational; it does not establish a rate that would survive more probes or more seeds.
What this means in practice
Deployment envelope, measured
- Best configuration. Qwen2.5-32B at 16-bit, 62.8% — open weights, but roughly 65 GB of them, so an 80 GB accelerator or two smaller cards.
- Smaller step. 14B at 16-bit reached 34.0% and fits a single 48 GB card.
- Tokens per task. 6,521 — the operator interface sends a plan and its results, not a corpus.
- If you must quantize, measure your quantized model on your own structured-planning tasks, not on general benchmarks. Our one 4-bit point scored below the 16-bit 32B model, and we cannot tell you whether that is the quantization or the model.
Secondary experiment: does more retrieval breadth help?
A fair criticism of our first writeup was that
the vector-RAG baseline retrieved one chunk only because its GPU was small.
On an H100 at full 32k context we re-ran it at the intended breadth of 20
chunks — a separate question from model scale, on a separate snapshot
(rag-breadth-h100), single run, same 94 tasks and model.
| vector-RAG configuration | corpus seen | tokens/task | accuracy |
|---|---|---|---|
| k=1 × 256-event chunks (24 GB card) | 0.4% | ~19k | 10.6% |
| k=20 × 24-event chunks (H100, 32k ctx) | 0.8% | 32,475 | 2.1% |
| TGMS, same run, same model | — | 6,521 | 36.2% |
Twenty chunks of 24 events is 480 of 59,835 — 0.8% of the corpus in total, not 0.8% at each of twenty sites. Widening retrieval lowered accuracy while multiplying the prompt roughly fivefold. Our reading is that spreading attention across twenty short, disconnected windows dilutes the ordering signal these questions quantify over: retrieving more serialised edges gives the model no explicit mechanism for composing a temporal join. Both retrieval cells are single runs and 2.1% is 2 tasks, so the ranking is safe while the exact ratio is not.
The number that did not move
Across every cell here, and two further model families on the same operator set, the number of emitted claims that failed verification against the execution trace was 0 — 0 of the 199 answers the gated system emitted across 282 task runs. Weaker models answered fewer questions rather than more of them wrongly — partly by declining to answer: in the primary campaign the gated system answered 70.6% of tasks (199 of 282) and was right 54.8% of the time on those. What ships is checked; not everything gets an answer.
What this result does not show
It does not show that quantization is the planning bottleneck. The 72B configuration changes two things at once — larger model, 4-bit weights — because that is the only way it fits a single 80 GB GPU, and one point moving two variables cannot separate them. We previously published that conclusion from this point; it was not supported and has been withdrawn. The controlled follow-up: the same model at 16-bit and at 4-bit, or quantization matched across every size. Until then the honest statement is that a 4-bit 72B model wrote worse plans than a 16-bit 32B model on this workload.
It does not show that the baselines are static. Text-to-Cypher improved roughly 2.9× in relative terms across the three 16-bit sizes. The measured claim is that it gained far less absolute accuracy and stayed well behind — not that a stronger model did nothing for it.
The sampling is thin, and the workload is one dataset. Every cell is a single run at one seed with no interval, unlike our primary campaign; three sizes establish an ordering, not a scaling law; the probe column rests on 13 questions. All of it is CollegeMsg. On a flat rating schema we have measured this interface and a SQL-generating baseline as a statistical tie.
Takeaway
Model capability and interface capability multiply rather than substitute. On this workload a bigger model paid about 4.5× under an interface that turns capability into computation, and far less under one that turns it into a longer prompt — a reason to measure your own interface before assuming the next model will fix it, and to treat any point that moves two variables at once as a question rather than a finding.
Evidence and reproduction
Frozen, SHA-pinned test splits against a store rebuilt byte-identically from its event log (tgms replay), on the University of Memphis
iTiger cluster. The scale table and the breadth study are in the
paper
under model and serving sensitivity; suites and harness are in
the repository, cut at
v0.3.0.
Every number here is quoted from docs/site_facts.json and
checked in CI.Continue reading
After this: what happens when the workload is written by people who have never seen the operator list — we let users define the workload.