The Improviser · Lesson 12 — The Big Band ← Course

The big band — mixture of experts.

One model can be many. A big band carries more players than any one tune needs — and on each note, only a couple of sections actually play.

Everything so far has been one player: a single network, every part of it working on every note. Make that player bigger and it knows more — but every note costs more too, because all of it has to move for each one. A mixture of experts breaks that trade. Split the network into sections — the experts — and let each note wake only a few of them.

The picking is done by a router: a small, learned part of the model that looks at the note in its context and scores every section. The top two play; the rest sit this one out. Next note, new scores, maybe a different pair. It isn't a bandleader pointing at the brass for the ballad — the router learned its cues in training, and the sections it groups together rarely line up with any label you'd give them.

That's what makes the band big and the bill small. All eight sections are on stage, and everything they know is in the model — that's the capacity. But each note only pays for two — that's the work. So a mixture-of-experts model can hold far more than a plain model that costs the same per note to run.

Two things it is not. The sections don't each play the tune and vote on an answer: a note is played once, by the few that were chosen, blended by how much the router trusted each. And sitting out isn't free — idle sections still have to be on stage, in memory, ready for the note that calls them.

Run one phrase through the band two ways. First the plain way — every section plays every note. Then through the router — two sections per note, chosen as the phrase goes. Watch who lights up, listen to the difference, and read the bill.

Pick how the band plays, then run the phrase. Start with every section to hear the plain model.
the phrase
the band · 8 sectionswaiting
work on this note0 of 8 sections

Score every section, wake two.

Inside one layer of the model, for one note. Nothing new about the sections themselves — the new part is the router in front of them, and the fact that most of them do nothing. 

# one layer of the band, for ONE note — x is the note in its context
scores = router(x)                      # a small learned net scores all 8 sections
chosen = top_k(scores, k=2)              # keep the best two; six sit this note out

y = 0
for s, w in chosen:                     # w: how much the router trusts section s
    y += w * sections[s](x)               # only these two do any work

# capacity on stage: 8 sections · work per note: 2

A huge band, a small combo on every note.

A mixture of experts splits parts of a model into sections and puts a learned router in front of them. For each note, the router scores every section and wakes only the top few — so the model carries the knowledge of the whole band while each note costs about what a small combo would. Which sections fire is decided note by note, from context, by cues the router learned in training.

Bigger without slower: every section on stage, two on the note. The router's choice — not a bandleader's, and not a vote — is what makes one model many.

Back to the course →
▸ Go deeper — where the experts actually live optional

Where the sections live

An expert is not a whole separate model. Inside each layer of a transformer, the block after the inner ear (Lesson 8) — the feed-forward part — is what gets split into experts, with a router in front. Attention is still shared by everyone. And the split is per layer, so a note's path is a different pair of sections in every layer it passes through.

The router and top-k

The router is tiny — often a single learned matrix. It turns the note's current state into a score per expert, softmaxes them, keeps the top k (two is common), renormalises those weights, and blends the chosen experts' outputs by them. It is trained together with everything else, with an extra nudge to spread the work around so no section ends up playing every note.

Parameters versus active parameters

This is why a mixture-of-experts model quotes two sizes. One widely used open model has about 47 billion parameters in total but touches only about 13 billion for any given note — eight experts per layer, two active. You pay roughly the compute of the smaller number and get the knowledge of the larger.

What it isn't, and what it costs

Experts aren't subject specialists. Probe them and you find groupings by the shape or position of a token far more often than by topic — and frequently nothing nameable at all. It isn't an ensemble either: the sections don't each answer and vote. And idle isn't free: every expert has to sit in memory, so the whole band still needs a big enough stage even though only two play.