The Third Umpire · Lesson 2 — The Working Memory ← Course

The working memory — the context window.

An LLM holds only so much on screen at once — whatever spills over is gone.

Upstairs, the third umpire studies a dismissal frame by frame — but the monitor holds only so many frames at a time. Feed in one more and the earliest scrolls off the top, gone as if it had never been shown.

An LLM works the same way. Whatever it can keep on screen while it answers is its context window, measured in tokens — the chunks of text from the last lesson. Anything inside the window it can use; anything past the edge might as well not exist.

How big is the screen? It depends on the model — from a few thousand tokens (a handful of pages) to over a million (a small library). But whatever the size, it is a fixed ceiling: there is always a last frame that fits, and a first one that falls off.

Add footage and watch the window fill, then overflow — then ask the umpire to decide, and see what it can still remember.

Only the last five frames are still on the screen — window = 5
Feed the umpire footage. Watch the window fill.
DRS · Working Memory window = 5
frames currently on screen
0 / 5 frames held · 0 fed in total
AWAITING DECISION

A fixed window. Oldest falls out.

Every frame is added to memory. When the window is full, the oldest frame falls out to make room — the umpire reasons only over what's left. 

WINDOW = 5            # the context window — fixed size
memory = []

def remember(frame):
    memory.append(frame)
    if len(memory) > WINDOW:
        memory.pop(0)     # oldest frame falls out of context

def decide():
    # the umpire can only reason over what's still in memory
    return "NOT OUT" if "front-foot" in memory else "OUT"

Inside the window, or gone.

You fed in more frames than the window could hold. Each new frame pushed the oldest one out. When the no-ball check dropped off, the umpire couldn't recall it — and called it wrong from what was left.

Every LLM works inside a fixed context window. Tokens inside it can be reasoned over; tokens pushed past the edge are gone. That's why long chats forget the start, and why what you keep in the window decides the answer.

Next: the shared budget →
▸ Go deeper — what the context window really is optional

Measured in tokens

The window is a fixed budget measured in tokens — the chunks from the previous lesson — not in words or sentences. Different models set the budget very differently, from a few thousand tokens to hundreds of thousands or more. Whatever the size, it's a hard ceiling: the model can attend to what's inside it and nothing else.

Everything shares the one budget

Crucially, everything competes for that same space: the standing instructions (the system prompt), any examples you provide, the entire back-and-forth so far, and the model's own reply as it's being written all draw from one budget. A long conversation doesn't get a bigger window — it just fills the one it has.

What “falling out” really means

When the total would exceed the window, the software around the model drops the oldest tokens before sending — the model simply never sees them. This isn't forgetting in any human sense — the text is simply no longer there to be read, the way a frame scrolled off the top of the screen can't be reviewed. That's why a long chat loses track of how it began, and why the start of a giant pasted document can quietly be ignored.

Why it matters when you build

Managing the window is part of the job: you decide what's worth keeping, what to summarise, and what to fetch only when needed (retrieval, later in the course). Position matters too — models tend to attend most strongly to the very start and the very end of a long window, and can skim what's buried in the middle.