The Video Assistant · Lesson 11 — The Referee's Report ← Course

The referee's report — measuring accuracy.

You can't trust — or improve — what you never measure. So score the calls against the truth.

After the match, the review team does not ask the VAR how the calls felt. It gathers a set of incidents whose true verdict is already settled — the replays leave no doubt — has the VAR rule each one again, and counts how many it got right. That count is the VAR's report card, and it is called an eval.

Swap the VAR for a model and it is the same idea. You take a set of questions whose right answers you already know — the ground truth — have the model answer each, and compare its answers against the truth. The fraction it gets right is its accuracy: the simplest score there is.

Why bother? Because “it seems good” is not a number. A real score lets you compare two prompts, catch a change that quietly made things worse, and decide whether the thing is fit to be trusted at all.

Run the referee's report over a set of appeals and watch the VAR's score come in.

A test set of appeals · the true call is known for each
Press play to grade the VAR against the replays.

A number you can trust and track.

The VAR got four of six right — a 67% report. Now that is something concrete: compare it to last week, to a different VAR, or to a new set of instructions, and you can see improvement or slippage instead of guessing at it.

An eval is a test set with known answers, the model's answers, and a score — and accuracy is the first and simplest one. The catch: a score is only as honest as the cases you put in the test, which is the next thing to get right.

Next: the two mistakes →
▸ Go deeper — what makes an eval trustworthy optional

Ground truth, and a held-out set

An eval needs cases whose right answer you already know. And you grade on a held-out set — incidents the model was not shown while being built. Score it on its own study notes and you measure memory, not skill.

Accuracy is not always enough

When a goal that should be ruled out is rare, a VAR who waves every goal through can score 90% and be useless — it never catches the ones that should not count. That is why other measures exist: how often its NO GOAL calls are right (precision) and how many of the real bad goals it catches (recall). Accuracy is the start, not the finish.

Judging answers that aren't right-or-wrong

Many outputs are not a clean call — a written explanation, a summary, a plan. Grading those needs a rubric and a human reviewer, or another model acting as judge against set criteria. Harder to score, but the same idea: compare against a standard.

Run it after every change

An eval earns its keep as a regression check: re-run the whole set after every prompt tweak or model swap. A change that fixes one case often breaks another, and only the score catches it before your users do.