The Improviser · Lesson 8 — The Long Thread ← Course

The long thread — reach and its limits.

Attention's real power is reach: it can lean on a note from far back as readily as the last one. But stretch the line out and the middle is where the listening goes faint.

The gift of attention is that it has no fixed reach. When the model picks its next note, it can lean just as directly on the very first note of the tune as on the last. That's how it resolves a phrase it opened long ago, or answers a question about the opening line of a long passage — reaching back and turning that distant note up loud.

But reach isn't the same as even reach. Across a very long line, attention isn't spread flat: it holds the opening and the most recent stretch clearly, while whatever sits in the middle comes through faint. Researchers call this being “lost in the middle”.

So a note that lands instantly at either end can be all but missed when it's buried halfway through — not because attention can't reach it, but because that's where the listening runs thinnest. It's a measured effect, not a metaphor: accuracy on a long input traces a U-shape, sharp at the edges and sagging in the middle.

Here's a long line — a callback note somewhere in it that the soloist needs to hear to resolve the tune. Slide it to different spots. Watch — and hear — how clearly it lands depending on where it sits.

The bars show how strongly attention reaches each spot in a long line. Move the callback note.
how clearly a note here is heard · 15-note line
openingmiddlemost recent
100%
callback note at position1 of 15
The high callback note rings out over a low pulse — loud where it's well heard, faint where it's lost.

Reach, and where it thins.

Attention can weight any earlier position — but on long inputs the weight it can spare for the middle is smaller. So where you put the thing that matters changes whether it's caught. 

# attention can reach ANY earlier position — no fixed limit
weight = attend(from=now, to=any_earlier_note)   # even note #1

# but over a long line the weight it spares isn't flat:
clear  = attend(to="opening")        # strong
clear  = attend(to="most recent")    # strong
faint  = attend(to="buried in the middle")  # weak — "lost in the middle"

# fix: put the key note at the start or the end, not mid-line

Long reach, uneven grip.

Attention's range is what lets a model hold a thread across a long passage — it can lean on the opening note as easily as the last. But that reach isn't uniform. On a long line the beginning and the recent end stay sharp while the middle goes soft, so a note buried mid-way can slip past even though nothing stops attention from reaching it.

Use it: in a long prompt, put the instruction or fact that matters most at the very start or the very end — not buried in the middle, where the model listens least.

Reach is the strength; the soft middle is the catch. Both come straight out of how attention spreads its weight across a long line.

Next: The Cost →
▸ Go deeper — reach, position, and the soft middle optional

Attention itself has no distance limit

Within the context window, every position can attend to every other in a single step — there's no “it fades with distance” built in, the way older recurrent models forgot the start of a sentence by the end. That direct, any-to-any reach is exactly what made the Transformer powerful.

So why does the middle get lost?

It's not one cause. Training data teaches models that beginnings and ends of documents carry weight; positional signals are cleaner near the edges; and the sheer number of competing middle positions splits the attention budget thin. The measured result, across many models, is a U-shaped curve: high accuracy for facts at the start or end of a long input, a dip in the middle.

The window edge is a harder limit

Reach only applies inside the context window from Lesson 2. Anything that has scrolled out of the window isn't faintly heard — it's simply gone, and no amount of attention brings it back. “Lost in the middle” is about faint; “out of the window” is about absent.

What people do about it

Practical fixes echo the tip above: put key material at the edges, keep prompts tighter, or retrieve just the relevant chunk (Lesson 7) instead of dumping a whole long document and hoping the middle gets heard. Newer long-context models push the dip down but rarely erase it.