The Video Assistant · Lesson 2 — The Working Memory ← Course

The working memory — the context window.

An LLM can only fit so much on screen at once — and the rest falls away.

The video assistant referee reviews a contested goal frame by frame, but the screen upstairs only holds so many frames at a time. Feed in one more and the earliest one scrolls off the top — gone, as if it had never been shown.

An LLM works the same way. The amount it can “keep on screen” while it answers is its context window, and it's measured in tokens — the little chunks of text from the last lesson. Everything inside the window the model can use; anything pushed past the edge is simply gone.

How big is the screen? It depends on the model — anywhere from a few thousand tokens (a handful of pages) to over a million (a small library). But however large, it is a fixed ceiling: there is always a last frame that fits, and a first one that falls off.

Add footage and watch the window fill, then overflow — then ask the VAR to decide, and see what it can still remember.

Only the last five frames are still on the screen — window = 5
Feed the VAR footage. Watch the window fill.
VAR · Working Memory window = 5
frames currently on screen
0 / 5 frames held · 0 fed in total
AWAITING DECISION

A fixed window. Oldest falls out.

Every frame is added to memory. When the window is full, the oldest frame falls out to make room — the VAR reasons only over what's left. 

WINDOW = 5            # the context window — fixed size
memory = []

def remember(frame):
    memory.append(frame)
    if len(memory) > WINDOW:
        memory.pop(0)     # oldest frame falls out of context

def decide():
    # the VAR can only reason over what's still in memory
    return "GOAL" if "onside" in memory else "NO GOAL"

Inside the window, or gone.

You fed in more frames than the window could hold. Each new frame pushed the oldest one out. When the onside check dropped off, the VAR couldn't recall it — and called it wrong from what was left.

Every LLM works inside a fixed context window. Tokens inside it can be reasoned over; tokens pushed past the edge are gone. That's why long chats forget the start, and why what you keep in the window decides the answer.

Next: the shared budget →
▸ Go deeper — what the context window really is optional

Measured in tokens

The window is a fixed budget measured in tokens — the chunks from the previous lesson — not in words or sentences. Different models set the budget very differently, from a few thousand tokens to hundreds of thousands or more. Whatever the size, it's a hard ceiling: the model can attend to what's inside it and nothing else.

Everything shares the one budget

Crucially, everything competes for that same space: the standing instructions (the system prompt), any examples you provide, the entire back-and-forth so far, and the model's own reply as it's being written all draw from one budget. A long conversation doesn't get a bigger window — it just fills the one it has.

What “falling out” really means

When the total would exceed the window, the software around the model drops the oldest tokens before sending — the model simply never sees them. This isn't forgetting in any human sense — the text is simply no longer there to be read, the way a frame scrolled off the top of the screen can't be reviewed. That's why a long chat loses track of how it began, and why the start of a giant pasted document can quietly be ignored.

Why it matters when you build

Managing the window is part of the job: you decide what's worth keeping, what to summarise, and what to fetch only when needed (retrieval, later in the course). Position matters too — models tend to attend most strongly to the very start and the very end of a long window, and can skim what's buried in the middle.