The Video Assistant · Lesson 12 — Coaching the Referee ← Course

Coaching the referee — fine-tuning vs prompting.

Two ways to change how the model behaves: instruct it every time, or retrain it for good.

You can change how the VAR performs in two very different ways. You can coach it before each match — a sharper briefing, a few example calls, the right footage and instruments to hand. Everything from the earlier lessons — system prompts, few-shot, retrieval, tools — is coaching: instructions given at the moment of the decision, changing nothing permanent.

Or you can retrain it — send it to the training ground for a season. Fine-tuning feeds the model thousands of example calls and nudges its internal weights, so new habits are baked in. The change is permanent and travels with the model, no instructions required.

Coaching is fast, cheap, reversible, and per-request; retraining is slow, costly, and changes the player itself. So the order matters: coach first, retrain only when coaching plateaus. And note the crucial catch — retraining changes behaviour and style, not knowledge: for fresh facts you still retrieve (RAG), you do not fine-tune.

For each situation, make the call — coach it, retrain it, or look it up — then see if you chose the right lever.

Scenario 1 of 5 · your call

Instruct first. Retrain only when you must.

Prompting and fine-tuning are two levers on the same model. Prompting shapes a single answer with instructions and context; fine-tuning reshapes the model's default behaviour by adjusting its weights on many examples. Reach for prompting first — it is faster, cheaper, and reversible. Fine-tune when you need a permanent, consistent behaviour or style that instructions can't reliably reach, or a narrow task where prompting has plateaued.

Keep the three levers straight: prompt to change behaviour now, fine-tune to change behaviour for good, retrieve to change what it knows. Most problems are solved long before you ever touch the weights — and that is the whole course: the same VAR, coached well.

Quick check
You need the model to answer in your club's exact house style across thousands of calls. The heaviest lever is…
Finish → the Honours Board
▸ Go deeper — what fine-tuning actually does optional

Adjusting the weights, not the prompt

Fine-tuning continues training on a dataset of your own input→output examples, nudging the model's weights by gradient descent so its default responses drift toward those examples. Unlike a prompt, the change is stored in the model — no instructions needed at run time, and it persists across every call.

It costs data, compute, and care

You need a quality labelled dataset (often hundreds to thousands of examples), the compute to train, and an eval (Lesson 11) to confirm it improved the target without regressing everything else. Fine-tune on too little or too narrow a set and the model can overfit — or “forget” skills it used to have.

It teaches form, not facts

Fine-tuning is for behaviour: tone, format, style, a specialised task. It is a poor way to add knowledge — the facts get frozen at training time, and training on facts can even increase hallucination. For anything that changes or must be current, retrieve it (RAG) instead.

Cheaper tuning — LoRA and adapters

Modern practice rarely retrains the whole model. Parameter-efficient methods like LoRA train a small set of add-on weights and leave the base frozen — far cheaper and faster, and you can keep several task-specific adapters for one base model. Same idea, a fraction of the cost.

The wider ladder

In practice you climb a ladder before fine-tuning: a better prompt → a few examples → tools and retrieval → and only then fine-tuning. Each rung is cheaper and more reversible than the next; fine-tuning is the top rung, not the first reach.