Attention, without a single equation

The problem it solves

Read this sentence: The trophy would not fit in the suitcase because it was too big.

What does “it” refer to? The trophy. Now change one word: ...because it was too small. Now “it” is the suitcase. Nothing about the position of the word changed; the meaning of a nearby word flipped the reference.

Any system that processes language has to solve this constantly. Attention is the mechanism that does it, and it is the single idea that made everything since 2017 possible.

What attention does, in plain terms

When the model is working on a particular token, it looks back over every earlier token and decides how much each one matters for this token, right now. Those relevance weights are computed, not fixed: they depend on the content.

Processing “it”, the mechanism assigns high relevance to “trophy” and “suitcase”, moderate relevance to “big”, almost none to “the”. The information from the relevant tokens is then blended into how “it” is represented. In effect, “it” becomes partly trophy-flavoured.

The model does this many times in parallel, with separate sets of weights called heads. Interpretability research has found individual heads that specialise: one tracks which noun a pronoun refers to, another tracks the subject-verb link, another tracks quotation boundaries. Nobody assigned these jobs. They fell out of training.

And the whole arrangement is stacked in layers (dozens of them) each refining the representation the last one produced. Early layers handle grammar-shaped relationships; later ones handle meaning-shaped ones.

Why it replaced what came before

The previous generation of language models read text strictly left to right, one word at a time, carrying a summary forward. Two things were wrong with that.

Information from early in a long passage had to survive being passed through hundreds of steps, and it faded. And because each step depended on the previous one, the work could not be split across processors: you cannot start word fifty before word forty-nine is done.

Attention fixes both at once. Every position can look directly at every other position, so nothing has to survive a long relay. And because all those lookups are independent, the whole computation can be done at once across thousands of processors. That is the sentence that explains the last eight years: attention made language models parallelisable, and parallelisable meant you could spend a hundred million pounds of computing on one and finish this year rather than next decade.

The paper is Attention Is All You Need, 2017, from a team at Google. It is freely readable on arXiv and it is short.

The cost, which you feel every day

If every token must be compared with every other token, then doubling the length quadruples the comparisons. Ten times the text is a hundred times the attention work.

This one property explains a great deal of what you experience:

• Long contexts are disproportionately expensive, which is why per-token prices rise in higher tiers and why very long documents are slow.
• Enormous engineering effort goes into cheaper approximations of attention, because whoever makes long contexts cheap changes the economics of the field.
• Context windows grew slowly at first (2,000 tokens, then 4,000, then 8,000) and then jumped to hundreds of thousands once the approximations improved.

The limitation: a window is not comprehension

A model advertising a million-token window can accept a million tokens. It does not follow that it uses them evenly.

A 2023 study, widely known as Lost in the Middle, tested models on finding a specific fact placed at different points in a long context. Accuracy was high when the fact sat near the beginning or the end, and fell substantially when it sat in the middle: a U-shaped curve. The effect has been reproduced repeatedly since, and it has softened in newer models without disappearing.

So when you paste a fifty-page contract and ask about clause 34 on page 27, the honest expectation is not that the model has read page 27 the way you would. It is that page 27 is available to be attended to, and attention is a competition it may lose.

What to do: put the important material at the start or the end, ask about one section at a time rather than the whole document at once, and quote the specific passage back into your question when a precise answer matters. All three work, and all three follow from the mechanism.