The Improviser · Lesson 1 — The Next Note ← Course

The next note — how a model decides.

It works like a musician taking a solo: choose the next note, then the next, then the next.

Watch someone improvise and it can look like pure invention. It isn't — but it isn't planned out in advance either. They build the line one note at a time, listening to what they've played so far — the key, the groove, where the phrase is heading — and each note they land narrows what should come next.

Some moments almost decide themselves: a phrase falling home to the tonic, the note the whole key leans on. Others stay wide open, where a dozen notes would all work. The player feels the difference and commits — firmly when one note clearly fits, more lightly when several do.

A large language model (or LLM) — the AI behind tools like ChatGPT — improvises the same way, only in words. It takes the text so far, weighs every possible next word against everything it absorbed in training, and plays one. Then it does it again, and again. It never writes the whole reply in advance — it builds it one piece at a time, which is why a chatbot types itself out in front of you instead of appearing all at once.

Set the feel and let the model play the next note — watch which notes it weighed, and hear what happens when many notes fit at once.

Pick a feel. The model weighs the notes that could come next, then plays one.
the line so far
Pick a feel to play the next note ↓

You hand over the notes. The model plays the next one.

You don't write the melody — you pass the notes so far and the feel, and ask. The model weighs it against every phrase it ever heard, and hands back a note and how sure it is. 

# Hand over the line so far — the model predicts the next note.
step = model.predict_next(
    played = ["C", "D", "E", "G"],
    key    = "C major",
    feel   = "resolve",
)

# step.note -> "C"   ·   step.confidence -> 0.74

Many options, one note.

Same four notes, three different feels. No single rule picked the next note — the model weighed every candidate against the key, the line and the style. When one note clearly led, it played with confidence. When many fit at once, it barely committed.

An LLM works the same way. It doesn't recall one right answer — it holds a set of odds over what could come next and plays the likeliest: confident when one option towers, hedged when they tie.

Quick check · your ear
The model puts most of the probability on one clear next note. That means it is…
Next: the odds →
▸ Go deeper — how the model actually chooses optional

It's a distribution, not a single note

When the model plays a note, it isn't pulling one answer from a shelf. Underneath, it assigns a probability to every note it could play next, and the note you hear is simply the most likely one. The confidence number in this lesson — 0.74 for the resolving phrase, 0.24 for the open one — is exactly that: the share of probability sitting on the top note. A tall, lonely spike means the model is sure; a near-even spread means it isn't, which is why "leave it hanging" comes back barely committed.

Weighed against training, not looked up

Where do those probabilities come from? From patterns the model absorbed across an enormous amount of music and text while training — the player's ear, baked into its weights. It is not looking up a particular song, and it has no specific recording it can replay. (Pulling an actual recording — a sample — is a separate capability called retrieval, and it gets its own lesson later.) This is the difference between an ear shaped by years of listening and cueing up the record.

Why "probabilistic" matters

Two consequences follow, and both become later lessons. Because the note is sampled from a distribution, the same phrase can resolve differently depending on how daring you let the model be — that's the temperature dial. And because the top note is merely the most likely, not the "right" one, a confident phrase can still be a cliché or a clam — that's the root of hallucination.