The Improviser · Lesson 6 — The Missing Doubt ← Course

The model can't feel the blank you'd feel.

Ask a person the chord under a bar they've never seen and they say "no idea." The model plays something confident and wrong. Here's the difference underneath.

To be fair, people invent too — we call it confabulation, misremembering, bluffing. But in ordinary factual talk we do it far less than a model does, and the reason why is the whole point of this page.

When you answer a question you're quietly doing two jobs at once. One is recall — reaching for the answer. The other is a check: do I actually know this? You feel the difference between "I've got it" and "I'm blank." When the chord isn't there, that second sense catches the gap and you say so. Knowing what you know is a separate faculty from the knowing itself.

The model only has the first job. It was trained to do exactly one thing: given the notes so far, produce the most plausible next one — and it always produces one, whether the answer is solid in its training or missing entirely. Nothing runs alongside to ask "do I know this?" So when the real answer isn't there, nothing stalls: the machinery fills the gap with whatever sounds most like an answer. Mechanically, a fluent fabrication is the exact same move as a correct recall.

That's why the model's confidence tells you nothing. In a person, doubt leaks out — you hedge, you pause. The model's fluency is produced independently of whether it's right, so a made-up chord rings out just as surely as a true one. Two things push it even further: training rewarded it for always giving an answer, and it has no way to check itself against the world — it plays from frozen memory, unable to glance at the real chart.

Ask the same impossible question of a person and of the model — watch the two pipelines run, and see exactly which step the model is missing.

Same question, no real answer behind it. Who should answer it?
the question
What chord sits under bar 17 of a tune neither has ever seen?
1 · recall — reach for the answer
↓
2 · the "do I actually know this?" check
↓
3 · what comes out

The model is missing the doubt, not the knowledge.

Both come up blank on a chord they never learned. The difference is the middle step. A person's check catches the blank and reports it, so an honest "I'm not sure" gets out. The model has no such check — the blank flows straight into "play something fluent," and out comes a confident invention.

So it isn't that the model knows less and lies more. It's that it lacks the layer of self-doubt you take for granted, and it was trained to always play something. That's exactly why the two fixes work: permission to say "not sure" hands back a piece of that missing check, and the chart removes the need to guess from memory at all.

Next: spot the phantom →
▸ Go deeper — "knowing that you know" optional

The name for the missing step

Psychologists call that second job metacognition — thinking about your own thinking, including a sense of how confident you should be. It's what lets you say "it's on the tip of my tongue" or "I'm sure of this one." A plain language model has no separate metacognitive read on its own answer; the words come out with a fluency that is unrelated to whether they're true.

Models can be given a bit of it

Newer systems bolt on rough substitutes: training that rewards calibrated uncertainty, a second pass where the model critiques its own draft, or a confidence score derived from the probabilities. These help, but none is the built-in gut check a person has — which is why grounding the model in real evidence remains the more reliable cure.

Why "always answer" was learned

During training a confident guess often scored better than a refusal, because a refusal is never right and a guess sometimes is. Over millions of examples the model learns that declining rarely wins — so it leans toward answering even when it shouldn't. Telling it, at request time, that "I don't know" is acceptable pushes back against that habit.