The Third Umpire · Lesson 1 — Frame by Frame ← Course

Frame by frame — tokens.

An LLM reads in chunks called tokens — sometimes a whole word, often only part of one.

A video umpire never takes in an incident at a single glance. It steps through the replay one frame at a time, and the full picture only assembles from that run of frames.

An LLM reads much the same way. Before it can do anything with your text, it splits it into tokens — small pieces, usually chunks of words. A common word like “the” is one token; a rarer or longer one is broken into several. The model never sees your letters, only this stream of chunks and the ID numbers behind them. It's also why a model can fumble something as simple as counting the r's in a word — it isn't looking at letters at all.

Take a sentence and break it into frames yourself — then see what the umpire actually reads.

An appeal is just text. Break it into frames.
"The snickometer caught a faint edge."

The same appeal, as the model takes it in.

You hand over text; the model first splits it into tokens. Most words are one token — but a rare word like snickometer becomes several. 

tokens = umpire.tokenize("The snickometer caught a faint edge.")
# → ['The', ' sn', 'ick', 'ometer', ' caught', ' a', ' faint', ' edge', '.']

len(tokens)     # 9 frames
tokens[1:4]    # [' sn', 'ick', 'ometer'] — one word, three tokens

Everything is tokens.

The appeal, the verdict, every word the umpire reads or speaks — all of it is a sequence of tokens. That's the only form the model ever handles.

Tokens are the unit an LLM works in — and the odds you met on the last page are odds over exactly these chunks, a fresh set for every frame the umpire adds. (How many tokens it can hold at once — its working memory — is Lesson 2.)

Next: the nets →
▸ Go deeper — the theory of tokens optional

What a token is

A token is the unit a language model actually operates on — not a single character, and not a whole word, but a chunk in between, usually a sub-word fragment. Every model carries a fixed vocabulary of these chunks (tens of thousands of them, on the order of 100,000–200,000 for recent models), and any text you send is cut into a sequence of them before the model does anything else. It never sees your sentence as letters or as meaning — it sees a list of token IDs.

Why sub-words, not letters or whole words

It's a trade-off. Work at the character level and every sentence becomes a very long sequence, and the model burns capacity just relearning how letters assemble into words. Work at the whole-word level and the vocabulary balloons, while anything unseen — a surname, a typo, new slang, a snippet of code — has no entry at all. Sub-word tokens thread the needle: frequent words like the, caught and edge each get a single token, while rarer or longer words are broken into familiar fragments the model already knows.

Why “snickometer” splits

That's why snickometer came apart into sn + ick + ometer — it's uncommon enough that the tokenizer never gave it a dedicated entry, so it's rebuilt from smaller pieces. The technique behind this, byte-pair encoding, builds its vocabulary by starting from raw bytes and repeatedly merging the most frequently co-occurring pair. Two quirks fall out of it: the leading space is usually part of the token (so edge and edge are different tokens), and a genuinely novel string falls back to tiny byte-level pieces rather than failing.

Why it matters when you build

Two things are measured in tokens, not words: the context limit — how much the model can hold at once (Lesson 2) — and the price you pay per call. As a rough feel, English averages about three-quarters of a word per token, but code, numbers and many non-English languages are far less efficient. Tokenization also explains some of an LLM's odd blind spots: because it sees sn / ick / ometer rather than individual letters, asking it to count the letters in a word or reverse a string is genuinely hard — it isn't looking at characters at all.