The maths — conditioning and diminishing returns.
The few ideas from probability that sit under few-shot — gathered in one place.
You need none of this to use few-shot; it is simply the machinery underneath, laid bare. And it rests on just two ideas: one from probability that explains why examples work at all, and one simple curve that explains why a few of them are plenty.
The first is conditioning — the model always answers “given” everything it has been shown, and examples are extra things to be given. The second is diminishing returns — reliability rises toward a ceiling while cost rises in a straight line.
Start with the curve: drag the number of examples and watch reliability climb toward its ceiling while the cost keeps rising straight past it.
Drag the examples. Green curves up to a ceiling; red rises in a straight line.
reliabilitycost (tokens)
n0
The formulas, in one place.
Each is written the way you would meet it in a textbook, with a plain-language line above and where it turned up in this lesson below.
Conditional probability
A conditional probability is the chance of one thing given that another is already true — the vertical bar reads “given”. The unlocking identity is P(A | B) = P(A and B) / P(B): you throw away every case where B is false and ask how often A happens in what is left. That is why conditioning on the right thing can swing a probability hard — it changes the pool of cases you count over. In general P(A | B) is not equal to P(A).
P(A | B) = P(A ∩ B) / P(B) — “A, given B”
P(A | B) ≠ P(A) in general — the condition moves the odds
P(A | B) ≠ P(A) in general — the condition moves the odds
ExampleAcross all deliveries, P(out) might be just 0.05. But given a faint sound as the ball passed the bat and the keeper taking it, P(out | sound, caught) could be 0.7. Same delivery; the condition changed the odds.
In Lesson 1: every next-token guess is conditional on the text so far.
Few-shot = extra conditioning
Zero-shot asks the model for P(answer | question). Few-shot hands it P(answer | examples, question) — the examples join the right-hand side of the bar as more “given” information. The question itself is unchanged; what changes is everything the model is told to assume. The illuminating part: relevant conditioning concentrates the distribution — it piles probability onto answers that fit the examples (the right format, the right kind of verdict) and starves everything else. That concentration is the steering you felt on Page 1.
zero-shot: P(y | x)
few-shot: P(y | e1, e2, …, ek, x) — examples join the condition
few-shot: P(y | e1, e2, …, ek, x) — examples join the condition
ExampleAsk the umpire “Caught behind?” on its own and you cannot guess the style of the reply — it might ramble, it might just say “Out.”, it might give a neat one-liner. Only about 1 in 5 replies land in one tidy format: a verdict, a short reason, then a confidence level — like “OUT · faint edge · high”. Now do it few-shot: first paste in two example appeals with their answers already filled in, in that same format, then ask your real one. About 9 in 10 replies now copy the format. Same question; the two worked examples made that style the likely answer.
In Lesson 4: Page 1 — the answer snapping into the demonstrated shape.
Which examples — ruling tasks out
There is a second way to read few-shot: the examples help the model infer which task you mean, out of all the tasks it could perform. By Bayes' rule the odds of a task update with evidence: P(task | examples) ∝ P(examples | task) · P(task). The key point is that an example only teaches something if it can rule a task out. Three OUT examples fit “always say out” and “actually judge the appeal” equally — they cannot tell them apart. A single NOT OUT example is impossible under “always say out”, so that reading collapses and the real task wins.
P(task | examples) ∝ P(examples | task) · P(task) — Bayes
ExampleSay the model might be doing “echo OUT”, “judge the appeal”, or “rate confidence”. Three clear OUTs are consistent with all three — no help. An OUT and a NOT OUT cannot come from “echo OUT”, so its probability drops to ~0 and “judge the appeal” takes over.
In Lesson 4: Page 3 — why boundary examples teach more.
Diminishing returns
Reliability does not climb in a straight line; it approaches a ceiling. Each example closes the same fraction of the gap that is left, so the absolute gain shrinks every time. That fixed-fraction rule is the unlocking identity — a geometric decay of the gap, gap(n) = (1 − f)·gap(n−1), which in closed form is an exponential approach. R0 is the zero-shot level, R∞ the ceiling few-shot can reach, and k how fast you get there.
gap(n) = R∞ − reliability(n) = (R∞ − R0)·e−k·n
reliability(n) = R∞ − (R∞ − R0)·e−k·n
reliability(n) = R∞ − (R∞ − R0)·e−k·n
ExampleWith R0 = 0.25, R∞ = 0.94, k = 0.8: n=1 → 0.63, n=2 → 0.80, n=3 → 0.88. The jumps are +38, +17, +8 points — each about 55% of whatever gap was left. By n=5 you are at 0.93, a whisker under the ceiling.
In Lesson 4: Page 2 — the reliability meter that shot up, then flattened.
Linear cost & where to stop
Every example is more text, so the token cost is a straight line: tokens(n) = base + per·n. Now put the two curves together. The benefit of one more example is its marginal gain — the shrinking step k(R∞−R0)e−k·n — while the marginal cost is constant, per tokens, every single time. Early on the gain dwarfs the cost; the sweet spot is the n where the next example's gain stops being worth its tokens. Past that you pay a straight-line cost for a benefit that has flattened to nothing.
tokens(n) = base + per·n — linear
gain(n) = (R∞−R0)·e−k(n−1)·(1−e−k) shrinks · cost step = per flat
gain(n) = (R∞−R0)·e−k(n−1)·(1−e−k) shrinks · cost step = per flat
Examplebase = 120, per = 60. Example 1 buys +38 points for 60 tokens; example 5 buys +2 points for the same 60. Same price, a fraction of the value — which is why two or three is usually where you stop.
In Lesson 4: Page 2 — the cost bar climbing while the meter stalled.
Where to take it next.
Each idea here is a doorway. If you want to keep going, these are the next words to search — each is the natural step up from what you just saw:
Bayes' theoremupdating the odds of a hypothesis as each example arrives — the engine behind “which task”.
In-context learningthe research name for a model learning from examples placed in the prompt, without any training.
Exponential decay & half-lifethe shape of the diminishing-returns curve — the same maths as cooling coffee or fading echoes.
Marginal analysisweighing the gain of one more against its cost — the economist's tool for “where to stop”.
None of it is needed to finish the course — but each deepens the picture of what few-shot is really doing.