The Third Umpire · Lesson 5 — The Other Dials ← Course

The other dials — top-k & top-p.

Temperature reshapes the whole spread of odds. These two dials instead cut the long shots out before the draw.

A real appeal has more than three possible calls — in a real model, thousands of possible next words, most of them nonsense. Temperature, last page, squeezed or stretched the whole set at once. Top-k and top-p are blunter: before the model draws, they throw the least likely options away entirely, so those simply cannot be picked.

Top-k keeps only the k most likely calls and bins the rest. Set k = 3 and the model may choose only among its top three; everything below that is gone.

Top-p (also called nucleus) is smarter: it keeps the fewest calls whose odds add up to at least p. Set p = 0.9 and it keeps just enough of the top calls to cover 90% of the probability — only a few when the model is confident, more when it is torn.

Either way, whatever survives is re-scaled so its odds add back up to 100%, and the draw happens only among those. The long tail of odd, unlikely calls can no longer sneak out.

Cut the tail with top-k or top-p and watch the unlikely calls drop away while the survivors share out the odds between them.

Tighten the dial. The long shots drop out; the survivors soak up the odds.
p0.90
→ the calls the draw may use (kept, re-scaled to 100%)

Two more arguments on the call.

Like temperature, these are just settings you pass. They are usually combined — temperature to flatten the odds, top-p to lop off the tail. 


      

    


    

      
Trim the tail, then draw.

      

        Temperature reshapes the odds; top-k and top-p cut the unlikely calls out before
        the draw — a fixed number of them (k), or as many as it takes to cover a share of
        the probability (p). Whatever survives is re-scaled to 100%, and the draw happens
        only there.
      

      

        The three often work together: temperature sets how flat the odds are, and
        top-p or top-k lop off the long tail — so the model stays varied without wandering
        into nonsense.
      

      

        Next: the maths →
        
      

    


  


  

    

      ▸ Go deeper — top-k vs top-p optional
      

        
k is blunt; p adapts

        

          Top-k always keeps the same number of options, even when the model is nearly
          certain (so k is too generous) or genuinely torn (so k is too strict). Top-p
          flexes with the moment — it keeps only a couple when the odds are peaked, and
          several when they are spread out. That adaptivity is why top-p is the more
          common default.
        

        
They ride alongside temperature

        

          These are not either/or with temperature. A typical setup is a middling
          temperature plus a top-p around 0.9–0.95: temperature controls how flat the
          odds are, top-p removes the unlikely tail. You are shaping the same distribution
          in two complementary ways.
        

        
Greedy, again

        

          Set top-k = 1 and only the single most likely call survives — the same
          deterministic behaviour as temperature 0. There are several roads to “always
          take the top pick”.
        

        
Why cut the tail at all

        

          Far down the odds sit strange, off-topic options with tiny probabilities. Leave
          them in and a high temperature will occasionally land on one, and the answer goes
          weird. Trimming the tail keeps the natural variety near the top while cutting the
          nonsense underneath.