The Improviser · Lesson 2 — The Maths ← Course
Optional · go further

The maths — tokens and attention.

The little bit of arithmetic under Lesson 2 — why a window fills on schedule, and why a longer one costs so much more than a short one.

You don't need any of this to keep going — if maths isn't your thing, skip ahead. But Lesson 2 quietly leaned on a few simple ideas, and they're worth seeing together: tokens add up, the take grows in a straight line, and — the interesting one — the cost of paying attention grows with the square of the length.

The first two are just addition. The third is the one worth slowing down for: it's where the real cost of a long window comes from — the part the lesson only hinted at.

Start with that third idea — drag the context length and watch the attention cost race ahead of the notes. Doubling the notes doesn't double the work.

Drag the context length. Notes grow in a line; the attention step grows with the square.
context length — n (tokens)
1,000
→ growth vs 1,000 tokens
notes · n×1×1
attention · n²×1×1

The formulas, in one place.

Each is written the way you'd meet it in a textbook, with a plain-language line above and where it turned up in this lesson below.

The context budget
Everything in a prompt is measured in one unit — tokens — so the parts simply add up, and the total can never exceed the window W. The useful move is to rearrange it with subtraction: the take you can keep is whatever's left after the fixed costs come off the top. Spend more of the window on a long brief or a long reply, and there's less room for the take. It's one budget, shared by all of it.
W ≥ S + H + M + R — window holds all four
Hmax = W − S − M − R — room left for the take
ExampleW = 2,000, brief = 180, request = 120, reply = 420. Then Hmax = 2000 − 180 − 120 − 420 = 1,280 tokens. At ~250 a turn, that's about 5 turns before the oldest must go.
In Lesson 2: The Shared Chart — the 2,000-token bar.
The take grows in a straight line
The take is just the running total of every turn so far — add a turn, add its tokens. The Σ symbol means "add them all up," and when the turns are about the same length k it collapses to plain multiplication. That straight-line growth is why you can predict exactly when the window fills, and why trimming arrives on a schedule rather than by surprise.
H(t) = Σi=1t ki — sum of all turns
H(t) ≈ k · t — when turns are equal-sized
Examplek = 250 tokens a turn, cap = 1,280. It fills when 250·t = 1,280, so t ≈ 5. Turn 6 pushes the oldest turn out of the window.
In Lesson 2: turns stacking up in the bar until one drops off.
Attention is quadratic — n²
To make sense of a passage, the model's attention step compares tokens pairwise. With causal masking, each token attends to itself and every earlier token: 1 + 2 + … + n = n(n+1)/2, about n²/2 — half the full n × n grid, but still growing with the square of n. So double the context and the attention step does roughly four times the work. That's a large part of why a longer window is expensive; the window's ceiling itself comes from how the model was trained and how much memory a long take costs to serve, not from this step alone.
comparisons ≈ n²/2 — with masking, each token attends to itself and the ones before it
1 + 2 + … + n = n(n+1)/2 — that exact count
attention(2n) / attention(n) ≈ 4 — 2× the notes, 4× the attention work
Examplen = 1,000 tokens → about 500,000 comparisons. Go to n = 2,000 → about 2,000,000 — twice the notes, four times the attention work.
In Lesson 2: why a longer window costs more.

Where to take it next.

Each idea above is a doorway. If you want to keep going, these are the next words to search — each is the natural step up from what you just saw:

Big-O notationthe shorthand for how cost grows with size — O(n) for the budget, O(n²) for attention.
Self-attentionthe mechanism whose every-token-sees-every-token wiring is where the n² comes from.
The KV cachethe standard trick that stops the model re-reading the whole take on every new token.
Sparse & linear attentionresearch aimed at beating the n² wall so windows can stretch — FlashAttention, sliding windows, and more.

None of these are needed to finish the course — but each turns "why is the window fixed?" into "here's how modern models stretch it anyway." That trail, from a simple sum to the cost of attention, is most of what separates a short context from a million-token one.

Quick check · your ear
You paste a very long brief and the model starts losing the earliest details. What ran out?
On to Lesson 3 →
▸ Go deeper — what this page does and doesn't cover optional

This is the cost maths

Everything here is about size and cost — how tokens add up into a budget, and why comparing every token with every other one grows with the square of the length. That's the part Lesson 2 leaned on, and it's most of what you need to reason about why a longer window costs so much more to run.

What's deliberately left out

It does not cover how attention actually decides what to attend to — the queries, keys and values inside self-attention. That's the mechanism under the n², and it's a deeper dive than this course takes.

If you do want to go there

The search terms are: self-attention, queries/keys/values, and the transformer architecture. That path explains not just the cost but the machinery that makes the whole window work at once.