{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"many-heads-and-what-they-do","title":"Many heads, and what has actually been found inside them","url":"https://addaly.com/learn/how-llms-work/many-heads-and-what-they-do","position":14,"of":78,"minutes":9,"answer":"Some attention heads have been shown to implement identifiable algorithms, most have not, and reading attention weights as an explanation of an output is a documented mistake.","keyIdea":"Some attention heads have been shown to implement identifiable algorithms, most have not, and reading attention weights as an explanation of an output is a documented mistake."},"body":{"format":"markdown","text":"## Why more than one\n\nA single attention head produces one set of weights per position: one opinion about where to look. That is limiting. A position often needs several things at once: the subject of its clause, the opening bracket it must close, the last time this entity was mentioned.\n\nRunning many heads in parallel, each with its own learned projections, lets the layer gather several different kinds of information in one step. Each head operates on a slice of the width: a model with width 4,096 and 32 heads gives each head 128 dimensions to work in. Total compute is roughly the same as one full-width head, so the parallelism is nearly free.\n\n## Heads that have actually been identified\n\nInterpretability research has found specific, verifiable circuits, mostly in small models studied deliberately. The best-established:\n\n**Previous-token heads.** Attend almost entirely to the position immediately before. Simple, and a necessary component of the next one.\n\n**Induction heads.** These implement a genuine algorithm: if the current token is `A`, find an earlier occurrence of `A`, and increase the probability of whatever followed it then. Formally, complete `[A][B] ... [A]` with `[B]`. This is how a model picks up on a made-up name or a pattern established earlier in your prompt, and it appears abruptly during training at the point where in-context learning ability jumps. It is the clearest link anyone has drawn between a mechanism and a capability.\n\n**Name-mover and inhibition heads.** In the sentence \"When Mary and John went to the shop, John gave a drink to ___\", researchers traced a circuit spanning several heads that identifies the candidate names, suppresses the one already used as the subject, and copies the other to the output. It was reverse-engineered in GPT-2 small and it holds up under intervention: knock out the inhibition head and the model produces the wrong name.\n\n**Copy-suppression heads.** These reduce the probability of tokens already present nearby, which appears to be part of how models avoid repeating themselves.\n\n## The honest caveats\n\nNow the corrections, which matter as much as the findings.\n\nMost heads have **no clean story**. Publications show the interpretable ones, which is a selection effect. A given head in a large model usually does several partial things at once.\n\nHeads are **redundant**. Removing a single head from a trained model often changes almost nothing, because several are doing overlapping work. This makes the \"one head, one job\" picture misleading even where the job is real.\n\nFindings from small models **may not transfer**. Most circuit-level results come from models with millions to a few hundred million parameters. Large models are studied with coarser tools, and the assumption that the same circuits are there in the same form is a hypothesis, not a result.\n\n## The mistake to avoid: attention as explanation\n\nYou will see attention-weight heatmaps presented as explanations of model behaviour: \"the model focused on the word *not*, which is why it classified the review as negative\". Treat this with suspicion. Two lines of research from 2019 showed that attention weights can often be substantially altered while leaving the output unchanged, and that different weightings can produce the same prediction. The debate that followed refined rather than overturned the point.\n\nThe reason is structural. Attention weights say where a value vector was gathered from. They say nothing about what the feed-forward layers then did with it, nothing about the residual stream contributions that bypassed attention entirely, and nothing about the other 31 heads in the same layer. A high weight is a plausible clue and not evidence.\n\n## What you can do with this\n\nIf you want to know whether a part of a model matters for a behaviour, the honest method is intervention, not observation: ablate it, or patch in activations from a different input, and see whether the output changes. That technique (activation patching) is the workhorse of the field and it is available free in open libraries such as TransformerLens, running on models small enough for a laptop.\n\nThe general principle transfers beyond interpretability. Watching where a system looks is weak evidence. Changing something and observing the difference is strong evidence."},"check":{"question":"A colleague shows an attention heatmap where the model's output position attends heavily to the word \"urgent\" and concludes this is why the ticket was classified as high priority. What is the strongest objection?","options":[{"text":"Attention weights are averaged across heads in the visualisation, which loses information.","correct":false,"tempting":"Averaging across heads really does obscure structure, and it is a legitimate criticism of many published heatmaps."},{"text":"The model may have attended to \"urgent\" without the word being present in the training data for that class.","correct":false,"tempting":"Training coverage is a habitual concern, though it does not bear on what the heatmap can show."},{"text":"Attention weights show where information was gathered from, not what was done with it; the classification depends on the feed-forward layers and residual contributions the heatmap does not display.","correct":true},{"text":"The heatmap shows the final layer only, and earlier layers may have attended elsewhere.","correct":false,"tempting":"Layer selection is a real limitation of these figures, so this identifies a genuine gap in the evidence, just not the central one."}]},"prev":{"slug":"the-block","title":"What one transformer block actually does","url":"https://addaly.com/learn/how-llms-work/the-block"},"next":{"slug":"the-feed-forward-half","title":"The feed-forward half, where most of the model is","url":"https://addaly.com/learn/how-llms-work/the-feed-forward-half"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Many heads, and what has actually been found inside them', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/many-heads-and-what-they-do (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Many heads, and what has actually been found inside them","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"Some attention heads have been shown to implement identifiable algorithms, most have not, and reading attention weights as an explanation of an output is a documented mistake.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-more-than-one","runs":[{"text":"Why more than one"}]},{"t":"p","runs":[{"text":"A single attention head produces one set of weights per position: one opinion about where to look. That is limiting. A position often needs several things at once: the subject of its clause, the opening bracket it must close, the last time this entity was mentioned."}]},{"t":"p","runs":[{"text":"Running many heads in parallel, each with its own learned projections, lets the layer gather several different kinds of information in one step. Each head operates on a slice of the width: a model with width 4,096 and 32 heads gives each head 128 dimensions to work in. Total compute is roughly the same as one full-width head, so the parallelism is nearly free."}]},{"t":"h2","id":"heads-that-have-actually-been-identified","runs":[{"text":"Heads that have actually been identified"}]},{"t":"p","runs":[{"text":"Interpretability research has found specific, verifiable circuits, mostly in small models studied deliberately. The best-established:"}]},{"t":"p","runs":[{"text":"Previous-token heads.","b":true},{"text":" Attend almost entirely to the position immediately before. Simple, and a necessary component of the next one."}]},{"t":"p","runs":[{"text":"Induction heads.","b":true},{"text":" These implement a genuine algorithm: if the current token is "},{"text":"A","code":true},{"text":", find an earlier occurrence of "},{"text":"A","code":true},{"text":", and increase the probability of whatever followed it then. Formally, complete "},{"text":"[A][B] ... [A]","code":true},{"text":" with "},{"text":"[B]","code":true},{"text":". This is how a model picks up on a made-up name or a pattern established earlier in your prompt, and it appears abruptly during training at the point where in-context learning ability jumps. It is the clearest link anyone has drawn between a mechanism and a capability."}]},{"t":"p","runs":[{"text":"Name-mover and inhibition heads.","b":true},{"text":" In the sentence “When Mary and John went to the shop, John gave a drink to ___”, researchers traced a circuit spanning several heads that identifies the candidate names, suppresses the one already used as the subject, and copies the other to the output. It was reverse-engineered in GPT-2 small and it holds up under intervention: knock out the inhibition head and the model produces the wrong name."}]},{"t":"p","runs":[{"text":"Copy-suppression heads.","b":true},{"text":" These reduce the probability of tokens already present nearby, which appears to be part of how models avoid repeating themselves."}]},{"t":"h2","id":"the-honest-caveats","runs":[{"text":"The honest caveats"}]},{"t":"p","runs":[{"text":"Now the corrections, which matter as much as the findings."}]},{"t":"p","runs":[{"text":"Most heads have "},{"text":"no clean story","b":true},{"text":". Publications show the interpretable ones, which is a selection effect. A given head in a large model usually does several partial things at once."}]},{"t":"p","runs":[{"text":"Heads are "},{"text":"redundant","b":true},{"text":". Removing a single head from a trained model often changes almost nothing, because several are doing overlapping work. This makes the “one head, one job” picture misleading even where the job is real."}]},{"t":"p","runs":[{"text":"Findings from small models "},{"text":"may not transfer","b":true},{"text":". Most circuit-level results come from models with millions to a few hundred million parameters. Large models are studied with coarser tools, and the assumption that the same circuits are there in the same form is a hypothesis, not a result."}]},{"t":"h2","id":"the-mistake-to-avoid-attention-as-explanation","runs":[{"text":"The mistake to avoid: attention as explanation"}]},{"t":"p","runs":[{"text":"You will see attention-weight heatmaps presented as explanations of model behaviour: “the model focused on the word "},{"text":"not","i":true},{"text":", which is why it classified the review as negative”. Treat this with suspicion. Two lines of research from 2019 showed that attention weights can often be substantially altered while leaving the output unchanged, and that different weightings can produce the same prediction. The debate that followed refined rather than overturned the point."}]},{"t":"p","runs":[{"text":"The reason is structural. Attention weights say where a value vector was gathered from. They say nothing about what the feed-forward layers then did with it, nothing about the residual stream contributions that bypassed attention entirely, and nothing about the other 31 heads in the same layer. A high weight is a plausible clue and not evidence."}]},{"t":"h2","id":"what-you-can-do-with-this","runs":[{"text":"What you can do with this"}]},{"t":"p","runs":[{"text":"If you want to know whether a part of a model matters for a behaviour, the honest method is intervention, not observation: ablate it, or patch in activations from a different input, and see whether the output changes. That technique (activation patching) is the workhorse of the field and it is available free in open libraries such as TransformerLens, running on models small enough for a laptop."}]},{"t":"p","runs":[{"text":"The general principle transfers beyond interpretability. Watching where a system looks is weak evidence. Changing something and observing the difference is strong evidence."}]}],"check":{"question":[{"text":"A colleague shows an attention heatmap where the model’s output position attends heavily to the word “urgent” and concludes this is why the ticket was classified as high priority. What is the strongest objection?"}],"options":[{"text":[{"text":"Attention weights are averaged across heads in the visualisation, which loses information."}],"correct":false,"tempting":[{"text":"Averaging across heads really does obscure structure, and it is a legitimate criticism of many published heatmaps."}]},{"text":[{"text":"The model may have attended to “urgent” without the word being present in the training data for that class."}],"correct":false,"tempting":[{"text":"Training coverage is a habitual concern, though it does not bear on what the heatmap can show."}]},{"text":[{"text":"Attention weights show where information was gathered from, not what was done with it; the classification depends on the feed-forward layers and residual contributions the heatmap does not display."}],"correct":true,"tempting":null},{"text":[{"text":"The heatmap shows the final layer only, and earlier layers may have attended elsewhere."}],"correct":false,"tempting":[{"text":"Layer selection is a real limitation of these figures, so this identifies a genuine gap in the evidence, just not the central one."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"the-block","title":"What one transformer block actually does","minutes":8},"next":{"slug":"the-feed-forward-half","title":"The feed-forward half, where most of the model is","minutes":9}}}