The Improviser · Lesson 12 — The Big Band ← Course

The big band — mixture of experts.

Bigger models know more. But in a plain model every part works on every note, so bigger always means slower and dearer. The big band is the way out.

Start with the problem. A bigger model is a better model — more knowledge, more skill — and the way to get one has always been to add more weights. But every player we've met so far has been a single network with all of it working on every note. Double the size and you double the cost of every word it produces: twice the memory, twice the arithmetic, twice the time and the electricity. Keep going and the model that knows the most is the one nobody can afford to run.

A mixture of experts is the trick that breaks the trade. Build the big model, but split much of it into sections, the way a big band is split into the trumpets, the trombones, the saxes, the rhythm section — groups of players, each with its own sound. Those groups are the experts; say eight of them, as in the band you're about to hear. And for each note, wake only two. The whole band is on stage, and everything it knows is still in there: that's the capacity. But each note is played by two sections, not all eight, so each note costs about what a small combo would: that's the work. A bigger brain without a bigger bill per word. That's why it's now the way nearly every large model is built.

Who decides which two? A router: a small, learned part of the model that looks at the note in its context and scores every section. The top two play; the rest sit this one out. Next note, new scores, maybe a different pair. It isn't a bandleader pointing at the brass for the ballad — the router learned its cues in training, and the sections it groups together rarely line up with any label you'd give them.

Two things it is not. The sections don't each play the tune and vote on an answer: a note is played once, by the few that were chosen, blended by how much the router trusted each. And sitting out isn't free — idle sections still have to be on stage, in memory, ready for the note that calls them. The bill per note shrinks; the size of the stage doesn't.

Run one phrase through the band two ways. First the plain way — every section plays every note. Then through the router — two sections per note, chosen as the phrase goes. Watch who lights up, listen to the difference, and read the bill.

Pick how the band plays, then run the phrase. Start with every section to hear the plain model.
the phrase
the band · 8 sectionswaiting
work on this note0 of 8 sections

Score every section, wake two.

Inside one layer of the model, for one note. Nothing new about the sections themselves — the new part is the router in front of them, and the fact that most of them do nothing. 

# one layer of the band, for ONE note — x is the note in its context
scores = router(x)                      # a small learned net scores all 8 sections
chosen = top_k(scores, k=2)              # keep the best two; six sit this note out

y = 0
for s, w in chosen:                     # w: how much the router trusts section s
    y += w * sections[s](x)               # only these two do any work

# capacity on stage: 8 sections · work per note: 2

A huge band, a small combo on every note.

The problem: a bigger model knows more, but in a plain model every weight works on every note, so bigger means slower and dearer per word. The fix: split parts of the model into sections and put a learned router in front of them. For each note the router scores every section and wakes only the top few — so the model carries the knowledge of the whole band while each note costs about what a small combo would. Which sections fire is decided note by note, from context, by cues the router learned in training.

Bigger without slower: every section on stage, two on the note. The router's choice — not a bandleader's, and not a vote — is what makes one model many.

Next: The Arrangement →
▸ Go deeper — where the experts actually live optional

Where the sections live

An expert is not a whole separate model. Inside each layer of a transformer, the block after the inner ear (Lesson 8) — the feed-forward part — is what gets split into experts, with a router in front. Attention is still shared by everyone. And the split is per layer, so a note's path is a different pair of sections in every layer it passes through.

The router and top-k

The router is tiny — often a single learned matrix. It turns the note's current state into a score per expert, softmaxes them, keeps the top k (two is common), renormalises those weights, and blends the chosen experts' outputs by them. It is trained together with everything else, with an extra nudge to spread the work around so no section ends up playing every note.

Parameters versus active parameters

This is why a mixture-of-experts model quotes two sizes. One widely used open model has about 47 billion parameters in total but touches only about 13 billion for any given note — eight experts per layer, two active. You pay roughly the compute of the smaller number and get the knowledge of the larger.

What it isn't, and what it costs

Experts aren't subject specialists. Probe them and you find groupings by the shape or position of a token far more often than by topic — and frequently nothing nameable at all. It isn't an ensemble either: the sections don't each answer and vote. And idle isn't free: every expert has to sit in memory, so the whole band still needs a big enough stage even though only two play.