The Improviser · Lesson 14 — Coaching the Player ← Course

Coaching the player — fine-tuning.

You've been writing "swing it" on the stand before every tune. It works, it costs every time, and it's still not who the player is. This lesson changes the player — a different kind of change, with a different bill.

Start with the itch. Say you want the player to swing, always, in the house style. So far you have one kind of lever, and it's the same lever every time: put it in front of the player. The brief (Lesson 3), a few worked examples (Lesson 4), recordings pulled from the library (Lesson 7) — all of it lands on the stand, in the working memory (Lesson 2), and the player reads it and plays. It works. But it costs room on the stand on every single phrase, it has to be there every single time, and the moment you take it away the player is exactly who it was. Nothing in the player moved. Its weights — the settled knowledge in its hands — were never touched.

Every working musician knows the other way. A session player who has spent years in swing bands doesn't need to be told; it's in the hands, and it comes out on every take with no instruction at all. Fine-tuning is how you get a model there. You take the player into the rehearsal room with a pile of examples of how you want it to play — hundreds, thousands — and run training again, the same way it learned in the first place: play, compare, nudge the weights, repeat. Musicians call that room the woodshed. Come out the other side and the habit is in the hands: it swings without being asked, on every tune, with nothing on the stand saying so.

That is what fine-tuning is for: behaviour that must be there every time — a voice, a format, a manner, a specialised way of reading a kind of input — and that you'd rather not spend room on the stand asking for on every phrase. It is not for facts. Drilling a player on a thousand phrases does not teach it tonight's new tune; it teaches it a way of playing. Facts still arrive on the stand, pulled from the library. And it does not make the player better in general — it bends the player toward the drills, and whatever the drills didn't cover can come out bent too. The specialist who's played nothing but swing for years has a Latin groove that's gone stiff. That is forgetting, and it is real.

So the order of reach is fixed. Ask first: a better brief, a few examples on the stand. Then hand it what it needs to know: the library. Only when the behaviour has to be baked in — and you have the examples, the compute, and a review (Lesson 13) to catch what the drilling broke — take the player into the room.

The player plays straight by default. You want swing. Get it two ways: put the ask on the stand, or take the player into the room and drill it. Hear each. Then, once it's coached, run the check-up — and listen for what the drilling cost.

Play the phrase as it comes. Then tick “swing it” on the stand and play again.
the stand · what's in the working memory8 tokens
brief— nothing asked —
from the librarytonight's tune
the askplay the phrase
the player · its weightsuntouched
rehearsal not run
the phrase
—

Three ways to get swing. Two touch the stand. One touches the player.

The first two build a context and leave the model alone. The third runs the training loop again on your examples and writes the result into the weights — which is why it lasts, costs, and can break things the examples never mentioned. 

# 1 · on the stand: the ask, every time (context only — weights untouched)
out = player("Swing it. " + phrase)

# 2 · on the stand, from the library: facts it doesn't have (context only)
pulled = library.lookup("tonight's tune")
out = player(pulled + phrase)

# 3 · in the room: fine-tuning (the weights themselves change)
for phrase, swung in drills:                 # thousands of (input, wanted output) pairs
    guess = player(phrase)
    loss  = distance(guess, swung)
    player.weights -= lr * grad(loss)     # the same nudge as training, on your data

out = player(phrase)                        # swings now, unasked — and may have lost other habits

Change the stand, or change the hands.

Prompting and retrieval put things in the context; the model reads them and stays as it was. Fine-tuning runs the training loop on your own examples and moves the weights, so the behaviour is there afterwards with nothing on the stand. That makes it the tool for durable style and manner — and a poor tool for facts, which still belong on the stand. It costs examples and compute, and it bends the model toward the drills, so habits the drills didn't cover can fade. Reach for the stand first; take the player into the room when the habit has to be in the hands, and review it afterwards.

Context is what the player reads. Weights are what the player is. Fine-tuning is the only tool here that changes the second — which is exactly why it's the last one you reach for.

Back to the course →
▸ Go deeper — what moves, what it costs, and the order of reach optional

What actually moves

Fine-tuning is the pretraining loop run again on a small, chosen dataset: forward pass, loss against the wanted output, gradients, a step on the weights. Full fine-tuning updates every weight. The cheaper and now more common form freezes the original model and trains small adapter matrices alongside it (LoRA and its relatives) — a few percent of the parameters — which are added to the frozen weights at inference. Either way, the result lives in the model, not in the prompt.

Why it holds behaviour and drops facts

A few thousand examples can shift how a model phrases, formats, refuses, or reads a specialised input, because those are patterns spread across many examples. A single fact appears once, and one exposure barely moves the weights — and if you repeat it enough to stick, you've started to overfit. For knowledge that changes or must be exact, retrieval into the context is more reliable, cheaper to update, and shows its sources.

Catastrophic forgetting

The weights that carry the new habit are the same weights that carried the old ones. Train on a narrow set and abilities outside it degrade — sometimes sharply. The usual guards are a low learning rate, mixing in general data, adapters rather than full updates, and a held-out eval suite run before and after. Fine-tuning without an eval is how regressions ship unnoticed.

The order of reach

Prompt engineering first: it is free, instant, and reversible. Then examples in the context. Then retrieval, when the model lacks information rather than manners. Fine-tune when the behaviour is stable, needed on every call, costly to spell out each time, and you have clean examples of it. It also earns its keep when it lets a smaller, cheaper model do a job that otherwise needed a larger one — the habit replaces a long prompt on every request.