The Improviser · Lesson 13 — The Review ← Course

The review — evals.

You changed the brief, or swapped the player. Did it get better? "Sounded good tonight" can't answer that. Only a count can — and counting is what an eval is.

Start with the problem. Every lesson so far gave you a lever — a better brief, a few examples, the right material on the stand — and every lever raises the same question: did it help? After a set, everyone has an opinion. The player sounded sharp; the player was off. None of that is a number, and none of it will tell you whether last week's change made things better or quietly worse. You can't trust, compare, or improve what you never measure.

Orchestras solved this long ago with the screened audition: every candidate plays the same fixed excerpts behind a curtain, and a panel scores each one against the same sheet. No reputation, no impression — just the excerpts and the marks. The review does the same to the player. It takes a stack of short phrases whose verdict is already settled — this one has a sour note (one note outside the key: what anyone would call a wrong note), this one is clean — plays each to the player, writes down the call, and compares it with the answer sheet. That stack, and the counting, is an eval. The stack is the dataset; the counting rule is the metric.

The first number out is accuracy: the share of calls that matched the sheet. It is honest and it is blunt, because there are two ways to be wrong and it lumps them together. Think of a smoke alarm. It can go off when you're only making toast — a false alarm — or stay silent while something is burning — a miss. The player's ear can do both: cry sour at a clean phrase, or wave through a phrase that really was sour. Sort every call by what the player said and what was true and you get four boxes, two right and two wrong. That grid is the confusion matrix, and it lets you ask two sharper questions. Precision: of the alarms it raised, how many were real? Recall: of the real sour notes, how many did it catch?

Those two pull against each other, and one control moves both: how picky the ear is — the alarm's sensitivity. Make it easygoing and it catches every sour note but cries wolf at clean ones — recall up, precision down. Make it picky and its alarms are always right, but it lets real sour notes through. There is no best setting in the abstract; it depends on which mistake costs more. A toast-triggered alarm is annoying; a silent one is a fire. F1 folds the two into one number that only stays high while both do.

And one warning the average hides. A player can score well on the whole stack and fail completely on one slice of it — every fast phrase, say — with the slow ones propping the number up. Cut the score by slice or you'll never know. Cut it, too, by where it was measured: a review is only as good as its stack. A stack the player has already rehearsed measures memory, not hearing; and a score earned in a quiet studio says little about a loud room in a different key.

Twelve phrases, six with a sour note. Tap any one to hear it. Set how picky the ear is, pick the room, then run the review: each phrase is played, called, and dropped into one of four boxes. Read the numbers — then cut them by slice.

Tap a phrase to hear it. Then run the review with the ear at its middle setting.
how picky the ear ismiddle · flags above 0.50
◂ easygoing — cries sour at littlepicky — needs to be sure ▸
the stack · 12 phrasesanswer sheet known
truly
sour
truly
clean
called
sour
0
caught it
0
false alarm
called
clean
0
missed it
0
rightly passed
—
precision
of the alarms, how many were real
—
recall
of the sour notes, how many caught
—
F1
the two, folded into one
—
accuracy
calls that matched the sheet — the blunt number

Play the stack, count the boxes.

Nothing about the player changes here. The eval is the loop around it: a set of phrases with known answers, a call per phrase, four running counts, and the numbers computed from them. 

# the review: a stack of phrases whose answer is already on the sheet
box = {"caught": 0, "false_alarm": 0, "missed": 0, "passed": 0}

for phrase, truth in answer_sheet:            # truth is "sour" or "clean"
    call = player.judge(phrase)               # the player's call, same words
    if   call == "sour"  and truth == "sour":  box["caught"]      += 1
    elif call == "sour"  and truth == "clean": box["false_alarm"] += 1
    elif call == "clean" and truth == "sour":  box["missed"]      += 1
    else:                                    box["passed"]      += 1

precision = box["caught"] / (box["caught"] + box["false_alarm"])   # alarms that were real
recall    = box["caught"] / (box["caught"] + box["missed"])        # sour notes that were caught
f1        = 2 * precision * recall / (precision + recall)

# then run the same loop on one slice at a time — fast phrases, slow phrases — and compare

Count, don't clap.

An eval is a dataset with known answers and a rule for scoring the model against them. Accuracy is the blunt score; the confusion matrix splits its errors into false alarms and misses, and from it come precision (are the alarms real?) and recall (are the real ones caught?). One control — how readily the model raises its hand — trades the two against each other, and F1 rewards keeping both up. Read the score by slice, because an average can hide a slice that fails outright; and read it against the stack it was earned on, because a rehearsed or too-tidy stack measures something other than the real room.

The answer sheet, the four boxes, the two sharp questions — and the two ways a good-looking number can lie: a hidden slice, and a stack that isn't the stage.

Next: Coaching the Player →
▸ Go deeper — the names, the formula, and why benchmarks drift optional

The four boxes, by their real names

Call the thing you're hunting for the positive — here, a sour note. Then caught it is a true positive, false alarm a false positive, missed it a false negative, and rightly passed a true negative. Precision is TP ÷ (TP + FP); recall is TP ÷ (TP + FN); accuracy is (TP + TN) ÷ everything. Which class counts as positive is your choice, and it flips the meaning of every box — so say it out loud before you quote a number.

F1 and the threshold

F1 is the harmonic mean of precision and recall: 2PR ÷ (P + R). Unlike a plain average it collapses when either one does — precision 1.0 with recall 0.2 gives F1 ≈ 0.33, not 0.6. The "how picky" slider is a decision threshold on a score the model produces; sweep it from one end to the other and you trace the whole precision–recall curve. Which point on that curve you ship is a product decision about which mistake is dearer, not a property of the model.

Slices and the hidden failure

Aggregate scores are dominated by whatever the dataset has most of. If fast phrases are a minority of the stack, the model can miss every one of them and still post a respectable overall recall. Real evals break results down by the axes that matter — length, language, topic, difficulty, the group a user belongs to — and treat the worst slice as the score that counts. This is also where most fairness problems are found.

Why a benchmark score drifts from real use

Two ways. Contamination: if the test phrases were in the training material, the model may be recalling answers rather than judging, and the score is inflated for reasons that won't survive new data. Mismatch: benchmarks are curated, tidy and fixed, while real inputs are messy, shifting, and often not what the benchmark measured at all. Multiple-choice questions in the studio; long, ambiguous requests on the stage. A published score is a floor for curiosity, not a ceiling for trust — the eval that matters is one built from your own room.