The Third Umpire · Lesson 2 — The Shared Budget ← Course

The shared budget — sharing the context window.

The window holds far more than your question — and it all has to fit at once.

Last page, the window held video frames, and the oldest fell out once it filled. A real conversation is no different — except the window has to hold far more than your latest question.

On every turn, four things share that one window: the standing instructions you gave the model (its “system prompt”), the whole chat so far, your new message, and the room it needs to write a reply. All four must fit together.

And the window never grows. A long chat simply fills it. Once it's full, the oldest parts of the conversation are dropped to make room — exactly like those frames sliding off the screen.

Add a few turns and watch the window fill. Keep going, and you'll see the earliest turn get pushed out.

Three voices — and every exchange piles into the same window.
System · the captain's briefing You are the third umpire. Give a one-line verdict.
Send the field umpire's appeal and see the verdict come back.
Add a turn. Watch the four parts share one fixed window.
one context window · 2,000 tokens
system prompt conversation so far your message reply (reserved)

It's all one list — and it all costs tokens.

Every turn, you send the model the same kind of list: the system prompt, the history, and your new message. The whole thing must fit the window, with room kept for the reply. 

messages = [
    {"role": "system", "content": briefing},   # set once, sent every turn
    *history,                                  # every past turn — this keeps growing
    {"role": "user", "content": new_message},  # your latest message
]

# the whole list must fit the window; trim the oldest history if it won't
reply = umpire.chat(messages, max_output_tokens=400)   # reply space, reserved up front

One window, shared by everything.

None of it gets special treatment. As the chat grows, it all eats into the same fixed window — and once that's full, the earliest turns fall out so the newest can fit.

So a big part of building with an LLM is choosing what to keep in the window — and what to drop, shorten, or look up only when you need it.

Next: does it remember you? →
▸ Go deeper — managing the budget optional

Reserve room for the reply

The model also needs space to write its answer, and that space is carved out of the same window before it begins. That's the max_output_tokens in the code above. Cram the window with input and you can leave too little room for the model to respond.

When it won't fit: trim, summarise, retrieve

Three moves keep a long conversation inside the window: drop the oldest turns (what you just saw); replace a long history with a short running summary; or store everything outside the window and fetch only the relevant bits on demand. That last one — retrieval — is powerful enough to earn its own lesson later.