The Improviser · Lesson 6 — When It Matters ← Course

Not every phantom is a disaster.

A made-up note can be a gift or a catastrophe — it depends entirely on what you're using the answer for.

It's tempting to conclude "the model makes things up, so never trust it." That's an overcorrection. The same invented chord that would ruin a chart the band reads on stage might be exactly the surprise you wanted jamming alone at midnight. The fabrication didn't change — the stakes did.

So the useful question is never "could this be wrong?" With a model, the answer is always yes. Ask instead: "what happens if it is?" When a wrong answer costs you nothing — or is even welcome — let the model run free. When it would mislead other people, or you can't easily take it back, slow down: ground it, cite it, and check.

Pick a task and see where it lands on the stakes meter — and what that means for how far you should trust the answer.

Same model, same chance of a phantom. What are you using it for?
low stakes · trust freelyhigh stakes · verify
—Pick a task above.
The same phantom risk means very different things depending on the job.
do—

Match the caution to the cost.

Hallucination isn't a reason to abandon the model or to trust it blindly — both are lazy. It's a reason to think for a second about the downside. Low downside: let it play, your judgment is the safety net. High downside: bring the chart, ask for the source, and verify before you rely on it.

That single habit — scaling your caution to what a wrong answer would cost — is what separates people who get burned by these tools from people who get real work out of them. And when the cost is high, the surest fix is to stop asking from memory and hand the model the real material: retrieval, the next lesson.

Next: the maths →
▸ Go deeper — a quick rule of thumb optional

Two questions, every time

Before you lean on an answer, ask: how likely is it to be wrong, and how bad is it if it is? A model raises the first probability across the board, so most of your judgment goes into the second. High cost and hard to undo — verify. Low cost or easily caught — proceed.

Reversible beats irreversible

A lot of the risk is about whether you can catch and fix the mistake in time. A wrong chord you play and immediately hear is self-correcting; a wrong fact printed in something others rely on is not. Treat one-way doors with far more care than things you can undo on the spot.

When in doubt, ground it

The safeguards from page one aren't only for high-stakes work, but that's where they earn their keep: letting the model say "not sure," handing it the real source to read from, and checking the answer against it. The next lesson is the big one — giving the model the actual material instead of asking it to remember.