{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"attention","title":"Attention, without a single matrix","url":"https://addaly.com/learn/how-llms-work/attention","position":10,"of":78,"minutes":8,"answer":"Attention blends information from earlier positions into the current one; it selects nothing and replaces nothing.","keyIdea":"Attention blends information from earlier positions into the current one; it selects nothing and replaces nothing."},"body":{"format":"markdown","text":"## The problem it solves\n\n\"The trophy did not fit in the suitcase because it was too small.\" Which is small? The suitcase. Now change \"small\" to \"large\": suddenly \"it\" is the trophy. A system that processes each word on its own cannot get this. The information that settles \"it\" sits several words away and flips depending on a word that has not been read yet at that point.\n\nAttention is the mechanism for moving information between positions. Here it is with no matrices.\n\n## Questions and adverts\n\nAt each position, from the vector currently sitting there, the model computes three smaller vectors:\n\n- a **query**: what this position is looking for\n- a **key**: what this position offers, an advert for its own contents\n- a **value**: the content it will hand over if anyone takes the advert\n\nNow take the position holding \"it\". Compare its query against the key of every position at or before it. The comparison is a dot product, which is large when two vectors point the same way. Those raw scores are converted into weights that sum to one. In our sentence, \"suitcase\" might receive 0.6, \"trophy\" 0.2, and the rest a little each.\n\nFinally, take the weighted average of the value vectors, in exactly those proportions, and add it into the vector at \"it\".\n\n> Figure: One attention step, at the position holding \"it\". Steps: 1. Form a query (From the vector sitting at this position: what is it looking for?); 2. Compare with every earlier key (One dot product per position, itself and everything before it, never after); 3. Turn the scores into weights (Suitcase 0.6, trophy 0.2, the rest a little each; they sum to one); 4. Average the values in those proportions (A weighted blend of what each earlier position offered); 5. Add the blend into the vector (Nothing was selected, nothing was replaced). Five operations and no decision anywhere in them. The pronoun was not resolved and nothing was recorded; the vector at \"it\" simply carries a suitcase-flavoured contribution from here on, and every later layer works with the enriched vector.\n\nThat is the whole operation. Notice what did not happen. Nothing was selected. The pronoun was not replaced. No decision was recorded. The vector at \"it\" now carries a suitcase-flavoured contribution, and every later layer works with that enriched vector.\n\n## Backwards only\n\nIn the models you use for generation, a position may attend to itself and to everything before it, never to what comes after. That restriction is the causal mask.\n\nIt has two consequences. During training, one pass over a document provides as many prediction problems as there are tokens, because each position is predicting its own successor without being able to peek. During generation, when the model writes token 40, tokens 41 onward do not exist yet.\n\n## Many heads, in parallel\n\nOne set of query, key and value comparisons is a head. A layer runs many of them side by side (32, 64, 128) each with its own learned projections, so each can compare on a different basis. One might track the immediately preceding token, another match an opening quotation mark to its close, another carry the subject of a sentence forward to its verb.\n\nBe careful with these stories. They are cleanest in small models studied deliberately, and they get blurry in large ones. Reading a head's attention weights as an explanation of why the model said something is a well-documented mistake: attention weights show where information was gathered from, not what caused the output.\n\n## The cost, which shapes the whole industry\n\nEvery position compares against every earlier position. Double the prompt and you quadruple the comparison work. A thousand tokens is about half a million pairs, per head, per layer. A hundred thousand tokens is about five billion.\n\nThat single quadratic loop is why long context was hard, why it was expensive before it became cheap, and why so much engineering (FlashAttention, sliding windows, grouped-query attention, key-value caching) targets this one operation and nothing else.\n\n## What attention is not\n\nAttention moves information between positions. It is not where the model's knowledge is stored. Most parameters, and much of whatever links \"Lagos\" to \"Nigeria\", live in the other half of each block, which the next lesson covers.\n\nA rough division to carry with you: attention decides what to look at, and the feed-forward layers decide what to do with what was found."},"check":{"question":"In \"The trophy did not fit in the suitcase because it was too small\", one head gives the position of \"it\" a weight of 0.6 on \"suitcase\" and 0.2 on \"trophy\". What does the model do with those weights?","options":[{"text":"It resolves the reference, choosing \"suitcase\" and substituting it for the pronoun before continuing.","correct":false,"tempting":"Attention is often described as resolving pronouns, and a clear 0.6 winner looks exactly like a decision being made."},{"text":"It adds a blend of those positions' value vectors, in those proportions, into the vector at \"it\", so later layers see an \"it\" carrying suitcase information.","correct":true},{"text":"The 0.6 is the model's confidence that its eventual answer will be correct.","correct":false,"tempting":"Any number between 0 and 1 inside a model invites being read as confidence about the output."},{"text":"It reorders the sequence so that related words sit next to each other for the next layer.","correct":false,"tempting":"\"Attention replaced recurrence, so position no longer matters\" is often heard as attention rearranging the sentence."}]},"prev":{"slug":"what-is-in-a-model-file","title":"What is actually inside a model you download","url":"https://addaly.com/learn/how-llms-work/what-is-in-a-model-file"},"next":{"slug":"attention-in-forty-lines","title":"Attention in forty lines of NumPy","url":"https://addaly.com/learn/how-llms-work/attention-in-forty-lines"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Attention, without a single matrix', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/attention (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Attention, without a single matrix","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"Attention blends information from earlier positions into the current one; it selects nothing and replaces nothing.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 2.1","figure":{"type":"flow","steps":[{"label":"Form a query","detail":"From the vector sitting at this position: what is it looking for?"},{"label":"Compare with every earlier key","detail":"One dot product per position, itself and everything before it, never after"},{"label":"Turn the scores into weights","detail":"Suitcase 0.6, trophy 0.2, the rest a little each; they sum to one"},{"label":"Average the values in those proportions","detail":"A weighted blend of what each earlier position offered"},{"label":"Add the blend into the vector","detail":"Nothing was selected, nothing was replaced"}],"title":"One attention step, at the position holding “it”","caption":"Five operations and no decision anywhere in them. The pronoun was not resolved and nothing was recorded; the vector at “it” simply carries a suitcase-flavoured contribution from here on, and every later layer works with the enriched vector."},"alt":"Figure: One attention step, at the position holding “it”. Steps: 1. Form a query (From the vector sitting at this position: what is it looking for?); 2. Compare with every earlier key (One dot product per position, itself and everything before it, never after); 3. Turn the scores into weights (Suitcase 0.6, trophy 0.2, the rest a little each; they sum to one); 4. Average the values in those proportions (A weighted blend of what each earlier position offered); 5. Add the blend into the vector (Nothing was selected, nothing was replaced). Five operations and no decision anywhere in them. The pronoun was not resolved and nothing was recorded; the vector at “it” simply carries a suitcase-flavoured contribution from here on, and every later layer works with the enriched vector."}},"blocks":[{"t":"h2","id":"the-problem-it-solves","runs":[{"text":"The problem it solves"}]},{"t":"p","runs":[{"text":"“The trophy did not fit in the suitcase because it was too small.” Which is small? The suitcase. Now change “small” to “large”: suddenly “it” is the trophy. A system that processes each word on its own cannot get this. The information that settles “it” sits several words away and flips depending on a word that has not been read yet at that point."}]},{"t":"p","runs":[{"text":"Attention is the mechanism for moving information between positions. Here it is with no matrices."}]},{"t":"h2","id":"questions-and-adverts","runs":[{"text":"Questions and adverts"}]},{"t":"p","runs":[{"text":"At each position, from the vector currently sitting there, the model computes three smaller vectors:"}]},{"t":"ul","items":[[{"text":"a "},{"text":"query","b":true},{"text":": what this position is looking for"}],[{"text":"a "},{"text":"key","b":true},{"text":": what this position offers, an advert for its own contents"}],[{"text":"a "},{"text":"value","b":true},{"text":": the content it will hand over if anyone takes the advert"}]]},{"t":"p","runs":[{"text":"Now take the position holding “it”. Compare its query against the key of every position at or before it. The comparison is a dot product, which is large when two vectors point the same way. Those raw scores are converted into weights that sum to one. In our sentence, “suitcase” might receive 0.6, “trophy” 0.2, and the rest a little each."}]},{"t":"p","runs":[{"text":"Finally, take the weighted average of the value vectors, in exactly those proportions, and add it into the vector at “it”."}]},{"t":"p","runs":[{"text":"That is the whole operation. Notice what did not happen. Nothing was selected. The pronoun was not replaced. No decision was recorded. The vector at “it” now carries a suitcase-flavoured contribution, and every later layer works with that enriched vector."}]},{"t":"h2","id":"backwards-only","runs":[{"text":"Backwards only"}]},{"t":"p","runs":[{"text":"In the models you use for generation, a position may attend to itself and to everything before it, never to what comes after. That restriction is the causal mask."}]},{"t":"p","runs":[{"text":"It has two consequences. During training, one pass over a document provides as many prediction problems as there are tokens, because each position is predicting its own successor without being able to peek. During generation, when the model writes token 40, tokens 41 onward do not exist yet."}]},{"t":"h2","id":"many-heads-in-parallel","runs":[{"text":"Many heads, in parallel"}]},{"t":"p","runs":[{"text":"One set of query, key and value comparisons is a head. A layer runs many of them side by side (32, 64, 128) each with its own learned projections, so each can compare on a different basis. One might track the immediately preceding token, another match an opening quotation mark to its close, another carry the subject of a sentence forward to its verb."}]},{"t":"p","runs":[{"text":"Be careful with these stories. They are cleanest in small models studied deliberately, and they get blurry in large ones. Reading a head’s attention weights as an explanation of why the model said something is a well-documented mistake: attention weights show where information was gathered from, not what caused the output."}]},{"t":"h2","id":"the-cost-which-shapes-the-whole-industry","runs":[{"text":"The cost, which shapes the whole industry"}]},{"t":"p","runs":[{"text":"Every position compares against every earlier position. Double the prompt and you quadruple the comparison work. A thousand tokens is about half a million pairs, per head, per layer. A hundred thousand tokens is about five billion."}]},{"t":"p","runs":[{"text":"That single quadratic loop is why long context was hard, why it was expensive before it became cheap, and why so much engineering (FlashAttention, sliding windows, grouped-query attention, key-value caching) targets this one operation and nothing else."}]},{"t":"h2","id":"what-attention-is-not","runs":[{"text":"What attention is not"}]},{"t":"p","runs":[{"text":"Attention moves information between positions. It is not where the model’s knowledge is stored. Most parameters, and much of whatever links “Lagos” to “Nigeria”, live in the other half of each block, which the next lesson covers."}]},{"t":"p","runs":[{"text":"A rough division to carry with you: attention decides what to look at, and the feed-forward layers decide what to do with what was found."}]}],"check":{"question":[{"text":"In “The trophy did not fit in the suitcase because it was too small”, one head gives the position of “it” a weight of 0.6 on “suitcase” and 0.2 on “trophy”. What does the model do with those weights?"}],"options":[{"text":[{"text":"It resolves the reference, choosing “suitcase” and substituting it for the pronoun before continuing."}],"correct":false,"tempting":[{"text":"Attention is often described as resolving pronouns, and a clear 0.6 winner looks exactly like a decision being made."}]},{"text":[{"text":"It adds a blend of those positions’ value vectors, in those proportions, into the vector at “it”, so later layers see an “it” carrying suitcase information."}],"correct":true,"tempting":null},{"text":[{"text":"The 0.6 is the model’s confidence that its eventual answer will be correct."}],"correct":false,"tempting":[{"text":"Any number between 0 and 1 inside a model invites being read as confidence about the output."}]},{"text":[{"text":"It reorders the sequence so that related words sit next to each other for the next layer."}],"correct":false,"tempting":[{"text":"“Attention replaced recurrence, so position no longer matters” is often heard as attention rearranging the sentence."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"what-is-in-a-model-file","title":"What is actually inside a model you download","minutes":8},"next":{"slug":"attention-in-forty-lines","title":"Attention in forty lines of NumPy","minutes":10}}}