The problem it solves
"The trophy did not fit in the suitcase because it was too small." Which is small? The suitcase. Now change "small" to "large": suddenly "it" is the trophy. A system that processes each word on its own cannot get this. The information that settles "it" sits several words away and flips depending on a word that has not been read yet at that point.
Attention is the mechanism for moving information between positions. Here it is with no matrices.
Questions and adverts
At each position, from the vector currently sitting there, the model computes three smaller vectors:
- a query — what this position is looking for
- a key — what this position offers, an advert for its own contents
- a value — the content it will hand over if anyone takes the advert
Now take the position holding "it". Compare its query against the key of every position at or before it. The comparison is a dot product, which is large when two vectors point the same way. Those raw scores are converted into weights that sum to one. In our sentence, "suitcase" might receive 0.6, "trophy" 0.2, and the rest a little each.
Finally, take the weighted average of the value vectors, in exactly those proportions, and add it into the vector at "it".
That is the whole operation. Notice what did not happen. Nothing was selected. The pronoun was not replaced. No decision was recorded. The vector at "it" now carries a suitcase-flavoured contribution, and every later layer works with that enriched vector.
Backwards only
In the models you use for generation, a position may attend to itself and to everything before it, never to what comes after. That restriction is the causal mask.
It has two consequences. During training, one pass over a document provides as many prediction problems as there are tokens, because each position is predicting its own successor without being able to peek. During generation, when the model writes token 40, tokens 41 onward genuinely do not exist yet.
Many heads, in parallel
One set of query, key and value comparisons is a head. A layer runs many of them side by side — 32, 64, 128 — each with its own learned projections, so each can compare on a different basis. One might track the immediately preceding token, another match an opening quotation mark to its close, another carry the subject of a sentence forward to its verb.
Be careful with these stories. They are cleanest in small models studied deliberately, and they get blurry in large ones. Reading a head's attention weights as an explanation of why the model said something is a well-documented mistake: attention weights show where information was gathered from, not what caused the output.
The cost, which shapes the whole industry
Every position compares against every earlier position. Double the prompt and you quadruple the comparison work. A thousand tokens is about half a million pairs, per head, per layer. A hundred thousand tokens is about five billion.
That single quadratic loop is why long context was hard, why it was expensive before it became cheap, and why so much engineering — FlashAttention, sliding windows, grouped-query attention, key-value caching — targets this one operation and nothing else.
What attention is not
Attention moves information between positions. It is not where the model's knowledge is stored. Most parameters, and much of whatever links "Lagos" to "Nigeria", live in the other half of each block, which the next lesson covers.
A rough division to carry with you: attention decides what to look at, and the feed-forward layers decide what to do with what was found.
Before you move on