The Video Assistant · Lesson 1 — Frame by Frame ← Course

Frame by frame — tokens.

An LLM reads in chunks called tokens — sometimes a whole word, often only part of one.

The VAR never takes in the whole incident in a single glance. It steps through the replay one frame at a time, and the full picture only forms from that sequence of frames.

An LLM reads in much the same way. Before it can do anything with your text, it first chops it into tokens — small pieces, usually chunks of words. A common word like "the" is a single token, while a rarer or longer word gets split into several. The model only ever sees this stream of tokens — never the raw letters, just the chunks and the numbers behind them.

Take a sentence and break it into frames yourself — then see what the VAR actually reads.

A check is just text. Break it into frames.
"The offside flag denied an equaliser."

The same call, as the model takes it in.

You hand over text; the model first splits it into tokens. Most words are one token — but a longer word like equaliser becomes several. 

tokens = var.tokenize("The offside flag denied an equaliser.")
# → ['The', ' offside', ' flag', ' denied', ' an', ' equ', 'al', 'iser', '.']

len(tokens)     # 9 frames
tokens[5:8]    # [' equ', 'al', 'iser'] — one word, three tokens

Everything is tokens.

The call, the verdict, every word the VAR reads or speaks — all of it is a sequence of tokens. The model never sees the raw letters; it sees these chunks, and the numbers behind them.

Tokens are the unit an LLM works in — and the odds you met on the last page are odds over exactly these chunks, a fresh set for every frame the VAR adds. (How many tokens it can hold at once — its working memory — is Lesson 2.)

Next: the training ground →
▸ Go deeper — the theory of tokens optional

What a token is

A token is the unit a language model actually operates on — not a single character, and not a whole word, but a chunk in between, usually a sub-word fragment. Every model carries a fixed vocabulary of these chunks (tens of thousands of them, on the order of 100,000–200,000 for recent models), and any text you send is cut into a sequence of them before the model does anything else. It never sees your sentence as letters or as meaning — it sees a list of token IDs.

Why sub-words, not letters or whole words

It's a trade-off. Work at the character level and every sentence becomes a very long sequence, and the model burns capacity just relearning how letters assemble into words. Work at the whole-word level and the vocabulary balloons, while anything unseen — a surname, a typo, new slang, a snippet of code — has no entry at all. Sub-word tokens thread the needle: frequent words like the, flag and offside each get a single token, while rarer or longer words are broken into familiar fragments the model already knows.

Why "equaliser" splits

That's why equaliser came apart into equ + al + iser — it's uncommon enough that the tokenizer never gave it a dedicated entry, so it's rebuilt from smaller pieces. The technique behind this, byte-pair encoding, builds its vocabulary by starting from raw bytes and repeatedly merging the most frequently co-occurring pair. Two quirks fall out of it: the leading space is usually part of the token (so flag and flag are different tokens), and a genuinely novel string falls back to tiny byte-level pieces rather than failing.

Why it matters when you build

Two things are measured in tokens, not words: the context limit — how much the model can hold at once (Lesson 2) — and the price you pay per call. As a rough feel, English averages about three-quarters of a word per token, but code, numbers and many non-English languages are far less efficient. Tokenization also explains some of an LLM's odd blind spots: because it sees equ / al / iser rather than individual letters, asking it to count the letters in a word or reverse a string is genuinely hard — it isn't looking at characters at all.