Query Advisor

The model proposes, the database disposes

Five questions, each answered by a statement that is correct and slow. The model gets the question, the slow statement, and three tools: the schema, the planner, and a read-only query runner. It proposes a rewrite and explains itself.

Then the part that is not a language model runs. Both statements execute against 1,000,000 orders and 1,500,001 order lines. Every cell of both result sets is hashed and compared. Both plans are captured with EXPLAIN (ANALYZE, BUFFERS). The verdict comes out of that comparison and nothing else.

Nothing here asks a second model to grade the first.

Why that matters more than it sounds

Nine in ten teams running agents in production have observability on them. About half have evaluations. The gap is not laziness — it is that agent output is usually prose, and the only judge available for prose is another model, so the measurement inherits the thing it was supposed to check.

SQL does not have that problem. Two statements either return the same rows or they do not, and one of them is either faster or it is not. This demo takes the one agent task where a deterministic oracle exists and builds the eval nobody can argue with.

There are four verdicts, and three of them are failures with different causes. Upheld is same rows, measurably faster. Wrong rows is a wrong answer. No faster is an honest rewrite that bought nothing. Invalid never ran. Collapsing those into a boolean would throw away the more interesting half of what was measured — a fast wrong answer and a correct pointless one are not the same finding.

What it measured

	Haiku 4.5	Opus 5
Revenue by status for a month	upheld, 12.3×	upheld, 8.1×
One customer’s order lines	no faster, 1.0×	upheld, 62.2×
Top ten customers	no faster, 0.6×	no faster, 0.8×
The value of twenty orders	upheld, 8.1×	upheld, 2,982×
Page 25,001 of the order list	upheld, 6,458×	upheld, 3,893×
	3 of 5 · $0.17	4 of 5 · $0.62

The last question was written to be the hard one: a keyset rewrite needs a cursor, and the cursor for that page is a row the model has to go and find. Both models found it. The hard one turned out to be the third, which is the sort of thing you only learn by running the suite rather than reasoning about it.

The question they both got wrong

Ten customers, ranked by order count, from a correlated count(*) evaluated once per candidate row. The textbook fix is to replace the correlated subquery with a grouped aggregate, and that is exactly what both models did.

Both got slower. 213ms became 329ms.

The baseline only ever counted the customers that matched the filter, using the index on customer_id. The rewrite grouped the entire million-row orders table first and filtered afterwards — strictly more work, arrived at by applying a rule that is usually right.

There is a rewrite that wins: filter the customers before aggregating, and it runs in 146ms. Neither model found it. And no amount of reading their explanations would have told you that, because both explanations are correct about why the original was slow. Only running both statements says which one is faster.

That question stays in the set exactly as it is. A golden set whose answer is always the one everybody already knows measures recall, not judgement.

The bug that mattered more than the scores

The first run reported both models as answering one question wrongly. They had not. The baseline for that question exceeded the reader role’s five-second statement budget, so it never ran — and the comparison, which checked the proposal and never the baseline, matched a real hash against an empty one and blamed the model.

A harness that blames the model for its own failure is worse than no harness, because it produces numbers that look like findings. The fix is a verdict of its own — the baseline failed — checked before anything is asked of the answer, and a question rewritten so its slow version is still ruinous and still finishes.

It took a query run by hand against the database to notice. That is the argument for running an eval against real data before believing what it prints, and it is why the scores above are worth reading: they survived the harness being wrong once.

Two roles that cannot do each other’s job

The model’s tools run as a role holding SELECT on five tables, read-only at the transaction level, with a five-second statement budget. The budget meter runs as a different role that can append to the spend ledger and cannot see the data at all.

A model that can reach the table recording what it costs is not being metered by it. CI asserts all four privileges rather than trusting the file that granted them — one GRANT is all it would take for this paragraph to become fiction.

The budget, and what happens when it runs out

This is a public endpoint that spends money when someone clicks it. Rate limiting caps how fast that happens, not how much of it there is, so there is a hard ceiling per day.

When the ceiling is reached the service refuses and says so. It does not serve a stored answer, for the same reason the SQL lab does not: the moment a visitor cannot tell a measurement from a recording, every number on the page stops being worth anything.