The Third Umpire · Lesson 11 — The Referee's Report ← Course

The referee's report — measuring accuracy.

You can't trust — or improve — what you never measure. So score the calls against the truth.

After the match, the referee does not ask the umpire how the calls felt. It gathers a set of deliveries whose true verdict is already settled — the replays leave no doubt — has the umpire rule each one again, and counts how many it got right. That count is the umpire's report card, and it is called an eval.

Swap the umpire for a model and it is the same idea. You take a set of questions whose right answers you already know — the ground truth — have the model answer each, and compare its answers against the truth. The fraction it gets right is its accuracy: the simplest score there is.

Why bother? Because “it seems good” is not a number. A real score lets you compare two prompts, catch a change that quietly made things worse, and decide whether the model is fit to trust at all.

Run the referee's report over a set of appeals and watch the umpire's score come in.

A test set of appeals · the true call is known for each
Press play to grade the umpire against the replays.

A number you can trust and track.

The umpire got four of six right — a 67% report. Now that is something concrete: compare it to last week, to a different umpire, or to a new set of instructions, and you can see improvement or slippage instead of guessing at it.

An eval is a test set with known answers, the model's answers, and a score — and accuracy is the first and simplest one. The catch: a score is only as honest as the cases you put in the test, which is the next thing to get right.

Next: the two mistakes →
▸ Go deeper — what makes an eval trustworthy optional

Ground truth, and a held-out set

An eval needs cases whose right answer you already know. And you grade on a held-out set — appeals the model was not shown while being built. Score it on its own study notes and you measure memory, not skill.

Accuracy is not always enough

When “out” is rare, an umpire who says not out to everything can score 90% and be useless — it never catches the wickets. That is why other measures exist: how often its “out” calls are right (precision) and how many real outs it catches (recall). Accuracy is the start, not the finish.

Judging answers that aren't right-or-wrong

Many outputs are not a clean call — a written explanation, a summary, a plan. Grading those needs a rubric and a human reviewer, or another model acting as judge against set criteria. Harder to score, but the same idea: compare against a standard.

Run it after every change

An eval earns its keep as a regression check: re-run the whole set after every prompt tweak or model swap. A change that fixes one case often breaks another, and only the score catches it before your users do.