We let users define the workload

In one sentence: we asked people who had never seen our operator list to write the questions they actually wanted answered, and only 10 of 110 could be expressed at all — a number that is 83 today, and that taught us more by missing its own forecast than by hitting it.

Why this matters

A benchmark written by the people who built the system will, without anyone intending it, tend to reflect the capabilities that system already has: you reach for examples you know are answerable, and the tasks that never occur to you are precisely the ones your design cannot express. The cheapest correction is to let somebody else write the questions, then publish the coverage number that comes back, whatever it is.

What we measured

What we found

All 110, disposed of completely — "10 expressible and 98 blocked" leaves two unaccounted for, and a reader is right to notice:

Nine percent, and every one of the ten was a multi-step chain rather than a single operation — the honest starting coverage of a thirteen-operation algebra against questions nobody steered.

Which capabilities were missing

Tagging each blocked question produces two very different counts, and running them together is the easiest mistake here. A capability can be one of several blockers, or the only thing in the way — and only the second tells you what shipping it would buy.

Three questions, in plain language

We built the top-ranked capability, and missed our own forecast

The histogram said grouped aggregation alone would unlock 30 questions — the whole argument for building it rather than something else. So the finished operator owed us a re-audit of all 110 against the new algebra, without touching the pre-registered table.

Fourteen moved but seventeen stayed blocked, which does not obviously add up to thirty, so here is the full accounting:

With grouping in hand, what the seventeen had been waiting for became legible: 11 want a set or pair join, 10 want ordered sequences inside a group, and 4 want both — 11 + 10 − 4 = 17, with no third group. Three also touch rating properties or division, but sit inside those counts rather than beside them.

None of this was concealed; it was folded into one tag by people who could see what was missing but not what sat underneath it. The histogram still ranked the work correctly — nothing else would have moved fourteen questions. But a number attached to a tag is an upper bound, not a forecast, and the fix is cheap: make every tag name the operation that would satisfy it, and the tags that cannot name one are the ones hiding a second capability.

That prediction has since been tested a second time, on the tag the histogram ranked next: arithmetic beyond counting — averages, medians, ratios, differences, percentages — was going to unlock 7 questions on its own. It unlocked 4, taking coverage to 28 of 110, and the shortfall has the same shape: of the questions that stayed blocked, most do not want a number computed from a result — they want a number computed per row of one, like the span between an account's first and last rating before those spans are averaged. Three more want a rank the operator cannot express, such as the top 1% of accounts. One tag, three capabilities, found the same way as the first: by building one of them and re-classifying everything it touched.

A third test has now run, on reading values out of the untyped property bag — the ratings themselves. It was going to unlock 13 questions; it unlocked 10, taking coverage to 38 of 110. Same shape a third time — what the tag was also carrying is the ability to hand a property's value back into a row, so that two rows can be compared with each other; an average reduces it and a filter tests it, but neither returns it. The forecasts are getting better as the tags get more specific, 47%, then 57%, then 77%, then 88% of what each predicted, and that is worth exactly as much as it sounds: the method of guessing from a tag has not improved, only the tags have. Whatever the next one predicts, discount it the same way.

What this result does not show

Coverage is not usefulness. Expressible means statable, not answered correctly; only 5 of the original ten could be scored verbatim, and how many a model gets right is a separate experiment.

It does not generalise past two datasets and one round. Both are public interaction logs. A different domain would produce a different histogram, and 110 questions from an unrecorded number of writers is a small, unstratified sample with no measure of annotation reliability.

The tags are judgments, not measurements — which is the finding rather than a caveat. The one tag we tested against reality overstated its delivery by more than two to one, and the roadmap inherits that uncertainty.

What this means in practice: the roadmap

Two measurement bases share that column, and the labels say which. Rows one and four to seven are July sole-blocker counts from the pre-registered table; rows two and three come from the August re-audit of the seventeen that stayed blocked, and they overlap by the four questions that want both, so they do not sum. Difficulty and composability are engineering judgment, not measurement, and the sole-blocker counts are exactly the quantity this study showed to be an upper bound. Read the ordering, not the arithmetic.

Seven rows have now been tested against reality, and the upper-bound finding held every time. Predicted-versus-delivered runs 30/14, 7/4, 13/10, 16/14, 15/15, 8/4, 5/5 and 5/4. The one exact hit in the middle came from the session that re-audited no blocked question at all, and each of the three sessions after it found tags still on the board for capabilities that had already shipped — so a count agreeing is not evidence the tags did.

The last two rows are the first forecasts made per question rather than in aggregate, and they came in 8 of 8 and 7 of 7 — including cells where reading the question contradicted its own tag. That is the only method we have found that predicts which questions move, and it is laborious: it means reading every blocked question again, not the label attached to it.

The last row also retired a tag outright. "Global scan-select" turned out to name three unrelated things — reading the store's own revision log, comparing two columns of one row, and finding the longest chronological chain. Two of the three shipped together and delivered exactly two questions each, which is what a forecast looks like once the label has been replaced by the questions underneath it.

One item did not come from the histogram at all. Four questions anchor transaction time to event years — asking what the database believed in 2013, when its beliefs begin at ingest — and seven target months outside the log entirely. Those writers had read the two-clock explanation. Storing two clocks is not enough; the interface has to teach them, and expose its coverage window.

Takeaway

Letting users define the workload cost us a flattering number and bought a ranked, measured roadmap — plus the more durable lesson: a capability histogram built before those capabilities exist ranks the work correctly while overstating what each item delivers, because a reader can only report the first thing that stops them.

Continue reading