The Improviser · Lesson 5 — The Other Dials ← Course

The other dials — top-k & top-p.

Temperature bends the odds without ever removing a note. These two dials do something blunter — they throw the long shots out before the draw, so they can't be played at all.

Last page, temperature squeezed or stretched the odds across every candidate note, but it never took one off the table. Top-k and top-p take the other route to keeping a solo from wandering: before the soloist draws, they delete the least likely notes outright.

Top-k keeps only the k most likely notes and drops the rest. Set k = 2 and the soloist may reach only for its top two notes; everything below that is gone.

Top-p (also called nucleus) is smarter: it keeps the fewest notes whose odds add up to at least p. Set p = 0.9 and it keeps just enough of the top notes to cover 90% of the probability — only a couple when the ear is sure, more when it is torn.

Here it is on the exact odds shown in the panel — C 40%, E 20%, G 14%, D 10%, E♭ 8%, A 5%, B♭ 3%. Walk down the list, adding as you go, and stop the moment the running total reaches p = 0.90: C 40 → 60 → 74 → 84 → 92% ✓ stop  ·  A and B♭ never counted So p = 0.9 keeps C, E, G, D, E♭ (together 92%) and bins the two long shots, A and B♭. Those five survivors are re-scaled to add back to 100%, and the draw happens only among them.

Why "smarter"? Because it counts probability, not notes. If the ear were dead sure — say C at 85%, E at 10% — the same p = 0.9 would keep just C and E (already 95%) and drop the rest. If the ear were torn, the odds spread thin across many notes, it would keep six or seven to reach the 90% mark. A fixed top-k can't flex like that: k = 2 always keeps exactly two, whether the model is certain or hopelessly split.

The payoff is the same either way: the long tail of odd, out-of-key notes is gone before the draw, so it can't sneak into the line — while the natural variety near the top survives untouched.

Cut the tail with top-k or top-p and watch the unlikely notes drop away while the survivors share out the odds between them — then play a few draws and hear the difference.

Tighten a dial. The long shots drop out; the surviving notes soak up the odds.
odds over the next note (kept notes re-scaled to 100%)
top-k 7
top-p 1.00

Two more arguments on the call.

Like temperature, these are just settings you pass. They are usually combined — temperature to flatten the odds, top-p or top-k to lop off the tail. 


      

    


    

      
Trim the tail, then draw.

      

        Temperature reshapes the odds; top-k and top-p cut the unlikely notes out before
        the draw — a fixed number of them (k), or as many as it takes to cover a share of
        the probability (p). Whatever survives is re-scaled to 100%, and the draw happens
        only there. Tight dials give a safe, in-key line; loose ones let the odd spicy
        note — an Eb or a Bb — slip through.
      

      

        In a real model these notes are tokens, and top-p — nucleus sampling — is the
        tail-trimmer most chat systems ship with, usually near 0.9. Temperature and the
        trimmer ride together: one sets how flat the odds are, the other decides how far
        down the list the draw may reach.
      


      

        
Quick check · your ear

        
The odds over the next note run C 40%, E 20%, G 14%, D 10%, then a thin tail. You set top-k = 2. Which notes can the next draw possibly play?

        

          
          
          
        

        

      


      

        Next: the maths →
        
      

    


  


  

    

      ▸ Go deeper — top-k vs top-p optional
      

        
k is blunt; p adapts

        

          Top-k always keeps the same number of notes, even when the ear is nearly certain
          (so k is too generous) or genuinely torn (so k is too strict). Top-p flexes with
          the moment — it keeps only a couple when the odds are peaked on one note, and
          several when they are spread across many. That adaptivity is why top-p, nucleus
          sampling, is the more common default.
        

        
They ride alongside temperature

        

          These are not either/or with temperature. A typical setup is a middling
          temperature plus a top-p around 0.9–0.95: temperature controls how flat the
          odds are, top-p removes the unlikely tail. You are shaping the same distribution
          in two complementary ways. When both a top-k and a top-p are set at once, a note
          has to survive both filters — it must be inside the top k and inside the
          nucleus — before it can be drawn.
        

        
Greedy, again

        

          Set top-k = 1 and only the single most likely note survives — the same
          deterministic playing as temperature 0. There are several roads to "always take
          the top note".
        

        
Why cut the tail at all

        

          Far down the odds sit strange, out-of-key notes with tiny probabilities. Leave
          them in and a high temperature will occasionally land on one, and the line goes
          sour. Trimming the tail keeps the natural variety near the top while cutting the
          clashing notes underneath.