{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"attention-what-it-does","title":"Attention, without a single equation","url":"https://addaly.com/learn/ai-actually-explained/attention-what-it-does","position":23,"of":92,"minutes":9,"answer":"Attention lets every token look directly at every other token, which made training parallel and huge models possible, and which costs work growing with the square of the length.","keyIdea":"Attention lets every token look directly at every other token, which made training parallel and huge models possible, and which costs work growing with the square of the length."},"body":{"format":"markdown","text":"## The problem it solves\n\nRead this sentence: *The trophy would not fit in the suitcase because it was too big.*\n\nWhat does \"it\" refer to? The trophy. Now change one word: *...because it was too small.* Now \"it\" is the suitcase. Nothing about the position of the word changed; the meaning of a nearby word flipped the reference.\n\nAny system that processes language has to solve this constantly. Attention is the mechanism that does it, and it is the single idea that made everything since 2017 possible.\n\n## What attention does, in plain terms\n\nWhen the model is working on a particular token, it looks back over every earlier token and decides how much each one matters *for this token, right now*. Those relevance weights are computed, not fixed: they depend on the content.\n\nProcessing \"it\", the mechanism assigns high relevance to \"trophy\" and \"suitcase\", moderate relevance to \"big\", almost none to \"the\". The information from the relevant tokens is then blended into how \"it\" is represented. In effect, \"it\" becomes partly trophy-flavoured.\n\nThe model does this many times in parallel, with separate sets of weights called **heads**. Interpretability research has found individual heads that specialise: one tracks which noun a pronoun refers to, another tracks the subject-verb link, another tracks quotation boundaries. Nobody assigned these jobs. They fell out of training.\n\nAnd the whole arrangement is stacked in layers (dozens of them) each refining the representation the last one produced. Early layers handle grammar-shaped relationships; later ones handle meaning-shaped ones.\n\n## Why it replaced what came before\n\nThe previous generation of language models read text strictly left to right, one word at a time, carrying a summary forward. Two things were wrong with that.\n\nInformation from early in a long passage had to survive being passed through hundreds of steps, and it faded. And because each step depended on the previous one, the work could not be split across processors: you cannot start word fifty before word forty-nine is done.\n\nAttention fixes both at once. Every position can look directly at every other position, so nothing has to survive a long relay. And because all those lookups are independent, the whole computation can be done at once across thousands of processors. That is the sentence that explains the last eight years: attention made language models **parallelisable**, and parallelisable meant you could spend a hundred million pounds of computing on one and finish this year rather than next decade.\n\nThe paper is *Attention Is All You Need*, 2017, from a team at Google. It is freely readable on arXiv and it is short.\n\n## The cost, which you feel every day\n\nIf every token must be compared with every other token, then doubling the length quadruples the comparisons. Ten times the text is a hundred times the attention work.\n\n> Figure: Why long contexts cost what they do. Line chart of Attention work, relative to 1,000 tokens against Tokens in the context: Comparisons between tokens (1000, 1) (2000, 4) (4000, 16) (8000, 64) (16000, 256). Every token is compared with every other, so the attention work grows with the square of the length. Double the text and it quadruples; ten times the text is a hundred times the work. This is the mechanism behind higher prices at longer contexts and behind the engineering effort spent on cheaper approximations.\n\nThis one property explains a great deal of what you experience:\n\n- Long contexts are disproportionately expensive, which is why per-token prices rise in higher tiers and why very long documents are slow.\n- Enormous engineering effort goes into cheaper approximations of attention, because whoever makes long contexts cheap changes the economics of the field.\n- Context windows grew slowly at first (2,000 tokens, then 4,000, then 8,000) and then jumped to hundreds of thousands once the approximations improved.\n\n## The limitation: a window is not comprehension\n\nA model advertising a million-token window can accept a million tokens. It does not follow that it uses them evenly.\n\nA 2023 study, widely known as *Lost in the Middle*, tested models on finding a specific fact placed at different points in a long context. Accuracy was high when the fact sat near the beginning or the end, and fell substantially when it sat in the middle: a U-shaped curve. The effect has been reproduced repeatedly since, and it has softened in newer models without disappearing.\n\nSo when you paste a fifty-page contract and ask about clause 34 on page 27, the honest expectation is not that the model has read page 27 the way you would. It is that page 27 is available to be attended to, and attention is a competition it may lose.\n\n**What to do:** put the important material at the start or the end, ask about one section at a time rather than the whole document at once, and quote the specific passage back into your question when a precise answer matters. All three work, and all three follow from the mechanism."},"check":{"question":"Why did attention make far larger language models practical, when the previous left-to-right designs did not?","options":[{"text":"It reduced the number of parameters needed, so models became cheaper to store and run.","correct":false,"tempting":"Reverses the effect; attention-based models are larger, not smaller, and their advantage is not parameter efficiency."},{"text":"It let the model read text in both directions, doubling the information available per word.","correct":false,"tempting":"Describes bidirectional reading, which some models do, rather than the property that changed training economics."},{"text":"It removed the need for training data to be labelled by people.","correct":false,"tempting":"Credits attention with self-supervised training on raw text, which predates it and is a separate idea."},{"text":"Every position can be computed independently rather than waiting for the previous one, so training can be spread across thousands of processors at once.","correct":true}]},"prev":{"slug":"the-next-word-game","title":"One chunk at a time, and it cannot take it back","url":"https://addaly.com/learn/ai-actually-explained/the-next-word-game"},"next":{"slug":"embeddings-meaning-as-position","title":"Meaning as a position in space","url":"https://addaly.com/learn/ai-actually-explained/embeddings-meaning-as-position"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Attention, without a single equation', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/attention-what-it-does (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Attention, without a single equation","courseTitle":"AI, Actually Explained","module":{"index":3,"slug":"how-a-chatbot-works","title":"What a chatbot is actually doing"},"answer":null,"keyIdea":"Attention lets every token look directly at every other token, which made training parallel and huge models possible, and which costs work growing with the square of the length.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Why long contexts cost what they do","series":[{"label":"Comparisons between tokens","points":[[1000,1],[2000,4],[4000,16],[8000,64],[16000,256]]}],"xLabel":"Tokens in the context","yLabel":"Attention work, relative to 1,000 tokens","caption":"Every token is compared with every other, so the attention work grows with the square of the length. Double the text and it quadruples; ten times the text is a hundred times the work. This is the mechanism behind higher prices at longer contexts and behind the engineering effort spent on cheaper approximations."},"alt":"Figure: Why long contexts cost what they do. Line chart of Attention work, relative to 1,000 tokens against Tokens in the context: Comparisons between tokens (1000, 1) (2000, 4) (4000, 16) (8000, 64) (16000, 256). Every token is compared with every other, so the attention work grows with the square of the length. Double the text and it quadruples; ten times the text is a hundred times the work. This is the mechanism behind higher prices at longer contexts and behind the engineering effort spent on cheaper approximations."}},"blocks":[{"t":"h2","id":"the-problem-it-solves","runs":[{"text":"The problem it solves"}]},{"t":"p","runs":[{"text":"Read this sentence: "},{"text":"The trophy would not fit in the suitcase because it was too big.","i":true}]},{"t":"p","runs":[{"text":"What does “it” refer to? The trophy. Now change one word: "},{"text":"...because it was too small.","i":true},{"text":" Now “it” is the suitcase. Nothing about the position of the word changed; the meaning of a nearby word flipped the reference."}]},{"t":"p","runs":[{"text":"Any system that processes language has to solve this constantly. Attention is the mechanism that does it, and it is the single idea that made everything since 2017 possible."}]},{"t":"h2","id":"what-attention-does-in-plain-terms","runs":[{"text":"What attention does, in plain terms"}]},{"t":"p","runs":[{"text":"When the model is working on a particular token, it looks back over every earlier token and decides how much each one matters "},{"text":"for this token, right now","i":true},{"text":". Those relevance weights are computed, not fixed: they depend on the content."}]},{"t":"p","runs":[{"text":"Processing “it”, the mechanism assigns high relevance to “trophy” and “suitcase”, moderate relevance to “big”, almost none to “the”. The information from the relevant tokens is then blended into how “it” is represented. In effect, “it” becomes partly trophy-flavoured."}]},{"t":"p","runs":[{"text":"The model does this many times in parallel, with separate sets of weights called "},{"text":"heads","b":true},{"text":". Interpretability research has found individual heads that specialise: one tracks which noun a pronoun refers to, another tracks the subject-verb link, another tracks quotation boundaries. Nobody assigned these jobs. They fell out of training."}]},{"t":"p","runs":[{"text":"And the whole arrangement is stacked in layers (dozens of them) each refining the representation the last one produced. Early layers handle grammar-shaped relationships; later ones handle meaning-shaped ones."}]},{"t":"h2","id":"why-it-replaced-what-came-before","runs":[{"text":"Why it replaced what came before"}]},{"t":"p","runs":[{"text":"The previous generation of language models read text strictly left to right, one word at a time, carrying a summary forward. Two things were wrong with that."}]},{"t":"p","runs":[{"text":"Information from early in a long passage had to survive being passed through hundreds of steps, and it faded. And because each step depended on the previous one, the work could not be split across processors: you cannot start word fifty before word forty-nine is done."}]},{"t":"p","runs":[{"text":"Attention fixes both at once. Every position can look directly at every other position, so nothing has to survive a long relay. And because all those lookups are independent, the whole computation can be done at once across thousands of processors. That is the sentence that explains the last eight years: attention made language models "},{"text":"parallelisable","b":true},{"text":", and parallelisable meant you could spend a hundred million pounds of computing on one and finish this year rather than next decade."}]},{"t":"p","runs":[{"text":"The paper is "},{"text":"Attention Is All You Need","i":true},{"text":", 2017, from a team at Google. It is freely readable on arXiv and it is short."}]},{"t":"h2","id":"the-cost-which-you-feel-every-day","runs":[{"text":"The cost, which you feel every day"}]},{"t":"p","runs":[{"text":"If every token must be compared with every other token, then doubling the length quadruples the comparisons. Ten times the text is a hundred times the attention work."}]},{"t":"p","runs":[{"text":"This one property explains a great deal of what you experience:"}]},{"t":"ul","items":[[{"text":"Long contexts are disproportionately expensive, which is why per-token prices rise in higher tiers and why very long documents are slow."}],[{"text":"Enormous engineering effort goes into cheaper approximations of attention, because whoever makes long contexts cheap changes the economics of the field."}],[{"text":"Context windows grew slowly at first (2,000 tokens, then 4,000, then 8,000) and then jumped to hundreds of thousands once the approximations improved."}]]},{"t":"h2","id":"the-limitation-a-window-is-not-comprehension","runs":[{"text":"The limitation: a window is not comprehension"}],"role":"limit"},{"t":"p","runs":[{"text":"A model advertising a million-token window can accept a million tokens. It does not follow that it uses them evenly."}]},{"t":"p","runs":[{"text":"A 2023 study, widely known as "},{"text":"Lost in the Middle","i":true},{"text":", tested models on finding a specific fact placed at different points in a long context. Accuracy was high when the fact sat near the beginning or the end, and fell substantially when it sat in the middle: a U-shaped curve. The effect has been reproduced repeatedly since, and it has softened in newer models without disappearing."}]},{"t":"p","runs":[{"text":"So when you paste a fifty-page contract and ask about clause 34 on page 27, the honest expectation is not that the model has read page 27 the way you would. It is that page 27 is available to be attended to, and attention is a competition it may lose."}]},{"t":"p","runs":[{"text":"What to do:","b":true},{"text":" put the important material at the start or the end, ask about one section at a time rather than the whole document at once, and quote the specific passage back into your question when a precise answer matters. All three work, and all three follow from the mechanism."}]}],"check":{"question":[{"text":"Why did attention make far larger language models practical, when the previous left-to-right designs did not?"}],"options":[{"text":[{"text":"It reduced the number of parameters needed, so models became cheaper to store and run."}],"correct":false,"tempting":[{"text":"Reverses the effect; attention-based models are larger, not smaller, and their advantage is not parameter efficiency."}]},{"text":[{"text":"It let the model read text in both directions, doubling the information available per word."}],"correct":false,"tempting":[{"text":"Describes bidirectional reading, which some models do, rather than the property that changed training economics."}]},{"text":[{"text":"It removed the need for training data to be labelled by people."}],"correct":false,"tempting":[{"text":"Credits attention with self-supervised training on raw text, which predates it and is a separate idea."}]},{"text":[{"text":"Every position can be computed independently rather than waiting for the previous one, so training can be spread across thousands of processors at once."}],"correct":true,"tempting":null}],"order":[2,0,1,3],"why":null},"prev":{"slug":"the-next-word-game","title":"One chunk at a time, and it cannot take it back","minutes":9},"next":{"slug":"embeddings-meaning-as-position","title":"Meaning as a position in space","minutes":10}}}