The Improviser · Lesson 8 — The Cost ← Course

The cost — why length gets expensive.

Attention's power comes from every note listening to every other. That same “everyone hears everyone” is exactly why a longer line costs so much more.

Here's the bill for that any-to-any reach. To choose the next note, each note in the line has to be weighed against every other note. Two notes: a couple of comparisons. Ten notes: every one checked against all ten. The work isn't the length of the line — it's the length times itself.

That's the shape people mean by “attention is quadratic”: n notes cost about n × n comparisons. It's like a room where everyone must shake hands with everyone — add a few more people and the number of handshakes jumps far faster than the headcount.

This is the hidden price behind the context window from Lesson 2. Doubling the prompt doesn't double the work — it roughly quadruples it. Attention is the piece that scales this way; the rest of the model grows only in step with the length, which is why context length, not raw model size, has been the wall — and why “just paste everything in” has a real ceiling.

Stretch the line and watch the grid of comparisons fill in — every note against every other. The count is the real cost, and it climbs far faster than the line gets longer.

Each square is one note weighed against another. Drag the line longer and watch the grid — and the count — swell.
every note (down) × every note (across) = every comparison
every note
every note
8
notes in the line
64
comparisons (n × n)
line length8 notes
double the notes → four times the work
8 notes
64
×4
16 notes
256

n notes, n × n comparisons.

Attention scores every position against every other — a full grid. That double loop is why the cost grows with the square of the length, not the length itself. 

# every note is weighed against every other note
for a in notes:            # n notes
    for b in notes:        # × n notes
        score[a][b] = relevance(a, b)

# total comparisons = n × n = n²
#   10 notes ->     100
#  100 notes ->  10,000
# 1000 notes -> 1,000,000   ← 10× the line, 100× the work

Power and price, same coin.

Letting every note hear every other note is what gives attention its reach — and it's the very thing that makes it expensive. The comparisons grow with the square of the line, so a prompt that's ten times longer is roughly a hundred times the work. That single fact sits behind why bigger context windows are hard, slow, and pricey.

Use it: shorter, well-aimed prompts aren't just tidier — they're cheaper and faster. Retrieving the few relevant chunks (Lesson 7) beats pasting a whole long document, because you pay for length squared.

Attention's genius and its bill are the same mechanism: everyone hears everyone. Great for holding a thread — costly the longer the thread gets.

Next lesson: Calling In a Guest →
▸ Go deeper — the quadratic wall and how people get around it optional

Why it's n² and not n

Attention builds a full n×n table of scores — each of the n positions against all n positions. Both the compute and the memory for that table grow with n². It's the defining cost of the Transformer, and the reason context length has been the hard frontier rather than an easy dial to turn up.

Caching helps generation

When a model writes token by token, it doesn't recompute everything each step — it caches the keys and values it already worked out (the “KV cache”). That speeds up generation, but the cache itself grows with the length, so long contexts still cost you in memory even with the trick.

Cheaper attention variants

A lot of research chases sub-quadratic attention: only letting each note attend to a nearby window, or to a sparse set of positions, or approximating the full table. These trade a bit of that any-to-any reach for a cost that grows closer to n. Every long-context model is some bargain of this kind.

The practical upshot

This is why providers price by tokens and why very long prompts feel sluggish and expensive. The mechanism that lets a model weigh the whole line at once is the same one that makes “the whole line” cost so much — so trimming and targeting your input pays off directly.