The Third Umpire · Lesson 12 — Coaching the Umpire ← Course

Coaching the umpire itself.

Every lesson so far handed the umpire a better briefing. This one changes the umpire.

There are two ways to change how the umpire calls a match. You can coach it at the start of each one — a sharper briefing, a few worked examples, the right footage to hand. Everything from the earlier lessons was coaching: instructions given at the moment of the decision, then gone. The umpire walks off unchanged.

Or you can retrain the umpire — a full season in the nets. Fine-tuning runs the model over thousands of example calls and nudges its internal weights, so the new habits are baked in. Nothing needs saying at match time; the change travels inside the model wherever it goes.

So the order is settled: coach first, retrain only once coaching plateaus. Retraining is slow and costly, and the umpire that comes back is a different player — worth it only when a briefing keeps falling short. And mind the catch: retraining changes how the umpire calls, not what it knows. For facts that shift week to week you still retrieve them (RAG); you never fine-tune them in.

For each situation, make the call — coach it, retrain it, or look it up — then see if you chose the right lever.

Scenario 1 of 5 · your call

Instruct first. Retrain only when you must.

Prompting and fine-tuning are two levers on the same model. Prompting shapes a single answer; fine-tuning reshapes the model's default by adjusting its weights over many examples — and the best of those examples come from human feedback on what a good call looks like, which is how the assistant models you use were taught to answer helpfully in the first place. Reach for prompting first: faster, cheaper, reversible. Fine-tune only when you need a behaviour or style to hold across every call, or a narrow task where the best prompts have plateaued.

Keep the three levers straight: prompt to change behaviour now, fine-tune to change behaviour for good, retrieve to change what it knows. Most problems are settled long before you touch the weights — and that is the whole course: the same umpire, coached well.

Finish → the Honours Board
▸ Go deeper — what fine-tuning actually does optional

Adjusting the weights, not the prompt

Fine-tuning continues training on a dataset of your own input→output examples, nudging the model's weights by gradient descent so its default responses drift toward those examples. Unlike a prompt, the change is stored in the model — no instructions needed at run time, and it persists across every call.

It costs data, compute, and care

You need a quality labelled dataset (often hundreds to thousands of examples), the compute to train, and an eval (Lesson 11) to confirm it improved the target without regressing everything else. Fine-tune on too little or too narrow a set and the model can overfit — or “forget” skills it used to have.

It teaches form, not facts

Fine-tuning is for behaviour: tone, format, style, a specialised task. It is a poor way to add knowledge — the facts get frozen at training time, and training on facts can even increase hallucination. For anything that changes or must be current, retrieve it (RAG) instead.

Cheaper tuning — LoRA and adapters

Modern practice rarely retrains the whole model. Parameter-efficient methods like LoRA train a small set of add-on weights and leave the base frozen — far cheaper and faster, and you can keep several task-specific adapters for one base model. Same idea, a fraction of the cost.

The wider ladder

In practice you climb a ladder before fine-tuning: a better prompt → a few examples → tools and retrieval → and only then fine-tuning. Each rung is cheaper and more reversible than the next; fine-tuning is the top rung, not the first reach.