The Improviser · Lesson 2 — The Working Memory ← Course

The working memory — the context window.

A model can only hold so much of the tune at once — and the rest quietly falls away.

A soloist stretching out over a long tune carries the recent bars in their head — where the line has been, the last turn of phrase, where it's heading. But only the recent ones. Play long enough and the opening slips away, as if it had never sounded.

A model works the same way. What it can "keep in its ear" as it plays is its context window, measured in tokens — the little chunks we met last lesson. Anything inside the window it can draw on; anything pushed past the edge is simply gone — not misremembered, gone.

How big is the window? It depends on the model — from a few thousand tokens (a page or two) to over a million (a whole songbook). But however large, the ceiling is fixed: there is always a last bar that still fits, and a first one that has just fallen off.

Play bars and watch the window fill, then overflow — then ask the model to echo the opening, and hear whether it can still remember it.

Play bars into the window. Watch it fill — then overflow.
working memory window = 5
bars currently in the ear
0 / 5 bars held · 0 played in total
nothing played yet

A fixed window. Oldest falls out.

Every bar is added to memory. When the window is full, the oldest bar falls out to make room — the model can only play off what's left. 

WINDOW = 5            # the context window — fixed size
memory = []

def hear(bar):
    memory.append(bar)
    if len(memory) > WINDOW:
        memory.pop(0)     # oldest bar falls out of context

def echo_opening():
    # can only call back to a bar still in memory
    return memory[0] if "Bar 1" in memory else "forgotten"

Inside the window, or gone.

You played in more bars than the window could hold. Each new bar pushed the oldest one out. Once the opening dropped off the edge, the model couldn't echo it — not because it chose not to, but because it was no longer there to play from.

That's the context window: a fixed span of recent tokens the model reasons over. Everything inside is fair game; everything past the edge is gone. Next we'll see that this window holds far more than the notes — and it all shares the same space.

Quick check · your ear
Once the opening bar has scrolled out of the window, why can't the model echo it?
Next: the shared chart →
▸ Go deeper — what "the window" really is optional

It's not memory in the human sense

The window isn't a memory the model stores and recalls — it's simply the stretch of tokens fed in on this turn, all present at once. Each time the model plays a note, it re-reads the whole window from scratch; nothing is "remembered" between turns except by being inside the text you send. Slide a token past the edge and it's not forgotten so much as never shown.

Why there's a ceiling at all

The window is fixed because the machinery that lets every token attend to every other token grows sharply with length — the cost rises with the square of the window, which is the arithmetic on the Maths page. Bigger windows are possible and keep getting bigger, but they are never free and never infinite.

What this means in practice

In a long session, the earliest things you said can quietly fall out of context — the model isn't ignoring them, it can no longer see them. The fix isn't a bigger memory; it's putting what matters back inside the window, or writing it down somewhere the model can be handed again. That's exactly where the next two pages go.