The Third Umpire · Lesson 5 — The Maths ← Course
Optional · go further

The maths — softmax and temperature.

How a single dial sharpens or flattens a set of scores into odds — the formula under all of Lesson 5.

None of this is required to use temperature — but it is the single formula sitting under all of Lesson 5, gathered in one place. And it is an unusually clean one: softmax, with one extra symbol dropped in (÷ T).

Softmax turns the model's raw scores into probabilities. Temperature divides those scores first, which stretches or squeezes the gaps between them. That is the whole mechanism — everything on Page 1 was this, and everything on Page 2 was a trim applied afterwards.

Drag the temperature and watch the three odds move — then read the formula that is doing it.

Same three scores every time. Temperature is the only thing changing the odds.
raw scores (logits), fixed — z
out2.0
not out1.4
inconcl.0.6
temperature
T1.0
plain softmax
→ probabilities · softmax(z / T)
out0%
not out0%
inconcl.0%

The formulas, in one place.

Each is written the way you would meet it in a textbook, with a plain-language line above and where it turned up in this lesson below.

Softmax
The model's raw scores — logits — can be any number, plus or minus, and are not probabilities. Softmax fixes that: exponentiate each score so they are all positive, then divide by the total so they sum to 1. The unlocking identity is that only the gaps between scores matter — pi/pj = ezi−zj — so adding the same amount to every score changes nothing.
pi = ezi / Σj ezj — scores → probabilities
pi / pj = ezi − zj — only the gap matters
ExampleScores 2.0, 1.4, 0.6. e2.0≈7.39, e1.4≈4.06, e0.6≈1.82; total ≈13.3 → 0.557, 0.306, 0.137. The out–not-out gap is 0.6, and 0.557/0.306 ≈ 1.82 = e0.6.
In Lesson 1: the engine behind every probability. Here: what temperature reaches into.
Temperature
Temperature divides every score by T before softmax runs. The unlocking point is what that does to the gaps: dividing the scores by T scales every gap by 1/T — the gap zi−zj becomes (zi−zj)/T. Below 1, T blows the gaps up, so the top score runs away and dominates; above 1, T shrinks the gaps, so the options even out. T = 1 is plain softmax, untouched.
pi(T) = ezi/T / Σj ezj/T
gap zi−zj → (zi−zj) / T — every gap scaled by 1/T
ExampleThe out–not-out gap is 0.6. At T = 0.5 it acts like 1.2, so the ratio is e1.2 ≈ 3.3 — out pulls away. At T = 2 it acts like 0.3, so the ratio is e0.3 ≈ 1.35 — nearly even. Same scores; T sets how sharply the top one is favoured.
In Lesson 5: Page 1 — the dial that made the umpire strict or wild.
The two extremes
Push T toward 0 and the biggest score's share races to 1 while the rest vanish — softmax becomes argmax, always the single top pick (greedy, deterministic). Push T toward infinity and every gap shrinks to 0, so the scores stop mattering and the odds flatten to uniform, 1/n each — a blind pick among all options. Everyday settings sit in between.
T → 0 : p → 1 for the top score, 0 for the rest (argmax)
T → ∞ : pi → 1/n for every option (uniform)
ExampleScores 2.0 / 1.4 / 0.6. At T = 0.1: out ≈ 99.8% — all but certain. At T = 10: about 36 / 33 / 31% — almost a three-way tie, the scores barely felt.
In Lesson 5: Page 1's “strict at zero, wild at the top”, made exact.
Trimming — top-k & top-p
The last dials do not touch the scores; they cut the distribution after softmax and renormalise. Pick a surviving set S — the k highest for top-k, or the smallest set whose probabilities reach p for top-p — zero out the rest, then divide each survivor by the survivors' own total so they add to 1 again. The unlocking idea is simply that renormalising means dividing by the mass you kept.
pi′ = pi / Σj∈S pj for i ∈ S, else 0 — keep S, re-scale to 1
ExampleOdds 0.45 / 0.25 / 0.15 / 0.08 / 0.05 / 0.02. Top-p 0.9 keeps the first four (they sum to 0.93). Renormalise: 0.45/0.93 = 0.48, 0.25/0.93 = 0.27, and so on — the survivors soak up the 7% that was cut.
In Lesson 5: Page 2 — the tail dropping to zero as the survivors grew.

Where to take it next.

Each idea here is a doorway. If you want to keep going, these are the next words to search — each is the natural step up from what you just saw:

The softmax / logistic functionthe same S-shaped map from scores to probabilities, everywhere in machine learning.
Entropya single number for how flat versus peaked a distribution is — raising temperature raises entropy.
The Boltzmann distributionphysics' version of this exact formula, temperature and all — where the name comes from.
Nucleus sampling (top-p)the idea behind Page 2's adaptive dial, from the paper that introduced it.

None of it is needed to finish the course — but each deepens the picture of what the temperature dial is really doing. And the same maths, borrowed straight from physics, turns up far beyond language models.

On to Lesson 6 →
▸ Go deeper — why the formula looks like this optional

Why exponentiate at all?

The exp does two jobs: it makes every score positive (so the results can be probabilities), and it turns adding scores into multiplying odds. That is exactly what makes “only the gaps matter” true, and it is the mirror image of the log-probabilities from Lesson 1, where multiplying probabilities became adding logs.

The numerical-stability trick

In practice you subtract the largest score from all of them before exponentiating: softmax(z) = softmax(z − c) for any constant c, because a shared constant cancels top and bottom. The answer is identical, but it keeps the numbers from blowing up. That is the m = max(...) you would see in real code.

Temperature is borrowed from physics

The name is not a metaphor invented for AI — it is lifted from the Boltzmann distribution in statistical physics, where a high temperature makes all states nearly equally likely and a low one pins the system into its lowest-energy state. Same formula, same intuition, a century older.

If you want to go there

The search terms are: softmax, the Boltzmann distribution, entropy, and nucleus sampling. The first three are one shared idea wearing different hats across ML, physics, and information theory.