{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"activation-patching-and-circuits","title":"Patching: finding which parts do the work","url":"https://addaly.com/learn/how-llms-work/activation-patching-and-circuits","position":72,"of":78,"minutes":9,"answer":"Copying one activation from a clean run into a corrupted run and measuring the recovery is the only method here that yields a causal claim.","keyIdea":"Copying one activation from a clean run into a corrupted run and measuring the recovery is the only method here that yields a causal claim, and a circuit is the set of components that patching shows to be sufficient and necessary for one narrow behaviour."},"body":{"format":"markdown","text":"## The question the first three lessons cannot answer\n\nA probe says a property is readable. A dictionary says a direction exists and what it fires on. Neither says whether the model *uses* it to produce the output in front of you, or where. For that you need to intervene, and the standard intervention is called activation patching.\n\n## The method\n\nChoose two prompts that differ in one thing and produce different answers. Clean: \"The Eiffel Tower is in the city of\": answer Paris. Corrupted: \"The Colosseum is in the city of\": answer Rome.\n\nRun both, and store every activation from the clean run. Now run the corrupted prompt again, but at **one** layer and **one** token position, overwrite the activation with the stored clean one. Measure how far the output moves back toward Paris.\n\nRepeat for every layer and every position, and you get a grid: where in the network does the clean information, inserted alone, recover the clean answer? Bright cells are places where the answer-relevant information is present and causally sufficient. Dark cells are places where it is not.\n\n```python\ndef patch(activation, hook, clean_cache, pos):\n    activation[:, pos] = clean_cache[hook.name][:, pos]\n    return activation\n# run the corrupted prompt with this hook at (layer, pos), read the Paris logit\n```\n\nThat is the whole method, and with TransformerLens on an open model it is about twenty lines. Variants patch individual attention heads or feed-forward blocks instead of whole layers; patch from corrupted into clean to test **necessity** rather than sufficiency; and approximate the whole grid at once with gradients (attribution patching) when there are thousands of components to test.\n\n> Figure: Activation patching, one cell at a time. Steps: 1. Build a pair differing in one thing (Clean: the Eiffel Tower is in the city of: Paris. Corrupted: the Colosseum: Rome.); 2. Run the clean prompt, store every activation (One cache, reused for the whole grid); 3. Run the corrupted prompt with one value overwritten (One layer, one token position, the stored clean activation in its place); 4. Measure how far the output moves back (Read the Paris logit: a bright cell means the information was present and sufficient); 5. Repeat for every layer and position (The grid is the map. Patch the other way to test necessity as well as sufficiency.). This is the only method in the module that yields a causal claim, which is why probes and dictionaries are checked against it. Two cautions it never loses: a corrupted prompt differing in several ways lights the grid up for all of them, and a removed component is often quietly covered for by another on the same forward pass.\n\n## What a circuit is\n\nA circuit is a subgraph of components (particular heads, particular feed-forward features, and the paths between them) that together implement one behaviour, verified by patching: keep only those components' contributions and most of the behaviour survives; remove only them and most of it goes.\n\nThe example module two mentioned is the clearest. \"When Mary and John went to the shop, John gave a drink to\": the answer is Mary, and the task is called indirect-object identification. In 2022 researchers patched their way through GPT-2 small and found around twenty-six heads in seven roles: heads that notice a name is duplicated, heads that suppress the duplicated one, heads that copy the remaining name to the output. Removing the circuit removed the behaviour. Keeping only the circuit kept most of it. It took months.\n\nOther circuits followed in small models: comparing years, completing docstrings, the induction behaviour from module two. Each is narrow, each is verified by intervention, and each is in a model far smaller than anything you use.\n\n## Automating the search\n\nDoing the indirect-object circuit by hand was not going to scale. From 2023, automated methods prune the model's full graph down to the edges that matter, using patching or its gradient approximation as the score. That cut circuit discovery in small models from months to hours.\n\nIn 2025 the approach was combined with the dictionaries of the previous lesson. Instead of heads and neurons, the nodes are dictionary features, and the method traces which features at one layer drive which at the next. Applied to a production-scale model, these attribution graphs gave the first mechanistic accounts of behaviours people care about: a model completing a two-step fact by passing through an intermediate concept, a model choosing a rhyme word several tokens before it reaches the end of the line and then writing toward it. The published caveat is that each graph explains a fraction of the behaviour, with the rest diffuse or in the dictionary's dark matter.\n\n## What \"explained\" means here\n\nA circuit is reported with a faithfulness figure. Eighty per cent means the circuit alone recovers eighty per cent of the difference in the answer's logit; the remaining twenty is spread across components too small to name. There is no circuit for \"answering questions\" or \"being helpful\". There are circuits for behaviours narrow enough to construct a clean-versus-corrupted pair for, and a mechanistic account of a model is a growing pile of those, not a single diagram.\n\n## Three things that go wrong\n\n**The baseline decides the map.** Patching measures the difference between two prompts. Choose a corrupted prompt that differs in several ways and the grid lights up for all of them at once. The art is in constructing pairs that differ in exactly one thing.\n\n**Self-repair.** Remove a head and, in many models, another head partly takes over its job on the same forward pass. Single ablations therefore understate how much a component matters, and a component that looks unimportant alone may be one of two that cover for each other.\n\n**Sufficiency is not the whole story.** A cell that recovers the answer when patched shows the information is there and usable. It does not show the model took that route unpatched. Sufficiency and necessity are both needed, and both are usually reported.\n\n## What you can do with it\n\nPatching is the only method in this module that yields a **causal** claim, and that makes it the ground truth against which probes and dictionaries are checked. In practice it is how you locate a behaviour before trying to change it, which is what the next two lessons do, first by pushing on a direction and then by editing a weight, and it is the reason you can trust the phrase \"this is where the fact lives\" when it appears."},"check":{"question":"Patching the clean activation into the corrupted run at layer 15, last-token position, restores 85 per cent of the correct answer's logit advantage. What has been shown?","options":[{"text":"That layer 15's weights store the fact, so editing those weights would change what the model believes about it.","correct":false,"tempting":"Confuses an activation (a value computed for this prompt) with the weights that computed it; where the fact is stored is a separate question the later editing lesson addresses."},{"text":"That the information needed for the answer is present at that layer and position and is causally sufficient there to recover most of the behaviour.","correct":true},{"text":"That the layers after 15 contribute nothing to this answer, since the correct information is already fully present by then.","correct":false,"tempting":"Later layers still have to read the patched information and turn it into the output; the patch works because they do that work, not because they are idle."},{"text":"That a linear probe at layer 15 would decode the answer with roughly 85 per cent accuracy, because patching and probing measure the same underlying quantity in different units.","correct":false,"tempting":"They measure different things (decodability versus causal effect) and a property can be highly decodable at a layer the model does not use, or used at a layer where a linear probe reads it poorly."}]},"prev":{"slug":"sparse-autoencoders","title":"Sparse autoencoders: a dictionary for the residual stream","url":"https://addaly.com/learn/how-llms-work/sparse-autoencoders"},"next":{"slug":"steering-with-directions","title":"Steering: pushing on a direction","url":"https://addaly.com/learn/how-llms-work/steering-with-directions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Patching: finding which parts do the work', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/activation-patching-and-circuits (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Patching: finding which parts do the work","courseTitle":"How a Language Model Actually Works","module":{"index":8,"slug":"looking-inside","title":"Looking inside: what the weights actually hold"},"answer":null,"keyIdea":"Copying one activation from a clean run into a corrupted run and measuring the recovery is the only method here that yields a causal claim, and a circuit is the set of components that patching shows to be sufficient and necessary for one narrow behaviour.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Build a pair differing in one thing","detail":"Clean: the Eiffel Tower is in the city of: Paris. Corrupted: the Colosseum: Rome."},{"label":"Run the clean prompt, store every activation","detail":"One cache, reused for the whole grid"},{"label":"Run the corrupted prompt with one value overwritten","detail":"One layer, one token position, the stored clean activation in its place"},{"label":"Measure how far the output moves back","detail":"Read the Paris logit: a bright cell means the information was present and sufficient"},{"label":"Repeat for every layer and position","detail":"The grid is the map. Patch the other way to test necessity as well as sufficiency."}],"title":"Activation patching, one cell at a time","caption":"This is the only method in the module that yields a causal claim, which is why probes and dictionaries are checked against it. Two cautions it never loses: a corrupted prompt differing in several ways lights the grid up for all of them, and a removed component is often quietly covered for by another on the same forward pass."},"alt":"Figure: Activation patching, one cell at a time. Steps: 1. Build a pair differing in one thing (Clean: the Eiffel Tower is in the city of: Paris. Corrupted: the Colosseum: Rome.); 2. Run the clean prompt, store every activation (One cache, reused for the whole grid); 3. Run the corrupted prompt with one value overwritten (One layer, one token position, the stored clean activation in its place); 4. Measure how far the output moves back (Read the Paris logit: a bright cell means the information was present and sufficient); 5. Repeat for every layer and position (The grid is the map. Patch the other way to test necessity as well as sufficiency.). This is the only method in the module that yields a causal claim, which is why probes and dictionaries are checked against it. Two cautions it never loses: a corrupted prompt differing in several ways lights the grid up for all of them, and a removed component is often quietly covered for by another on the same forward pass."}},"blocks":[{"t":"h2","id":"the-question-the-first-three-lessons-cannot-answer","runs":[{"text":"The question the first three lessons cannot answer"}]},{"t":"p","runs":[{"text":"A probe says a property is readable. A dictionary says a direction exists and what it fires on. Neither says whether the model "},{"text":"uses","i":true},{"text":" it to produce the output in front of you, or where. For that you need to intervene, and the standard intervention is called activation patching."}]},{"t":"h2","id":"the-method","runs":[{"text":"The method"}]},{"t":"p","runs":[{"text":"Choose two prompts that differ in one thing and produce different answers. Clean: “The Eiffel Tower is in the city of”: answer Paris. Corrupted: “The Colosseum is in the city of”: answer Rome."}]},{"t":"p","runs":[{"text":"Run both, and store every activation from the clean run. Now run the corrupted prompt again, but at "},{"text":"one","b":true},{"text":" layer and "},{"text":"one","b":true},{"text":" token position, overwrite the activation with the stored clean one. Measure how far the output moves back toward Paris."}]},{"t":"p","runs":[{"text":"Repeat for every layer and every position, and you get a grid: where in the network does the clean information, inserted alone, recover the clean answer? Bright cells are places where the answer-relevant information is present and causally sufficient. Dark cells are places where it is not."}]},{"t":"code","lang":"python","label":"Python","text":"def patch(activation, hook, clean_cache, pos):\n    activation[:, pos] = clean_cache[hook.name][:, pos]\n    return activation\n# run the corrupted prompt with this hook at (layer, pos), read the Paris logit"},{"t":"p","runs":[{"text":"That is the whole method, and with TransformerLens on an open model it is about twenty lines. Variants patch individual attention heads or feed-forward blocks instead of whole layers; patch from corrupted into clean to test "},{"text":"necessity","b":true},{"text":" rather than sufficiency; and approximate the whole grid at once with gradients (attribution patching) when there are thousands of components to test."}]},{"t":"h2","id":"what-a-circuit-is","runs":[{"text":"What a circuit is"}]},{"t":"p","runs":[{"text":"A circuit is a subgraph of components (particular heads, particular feed-forward features, and the paths between them) that together implement one behaviour, verified by patching: keep only those components’ contributions and most of the behaviour survives; remove only them and most of it goes."}]},{"t":"p","runs":[{"text":"The example module two mentioned is the clearest. “When Mary and John went to the shop, John gave a drink to”: the answer is Mary, and the task is called indirect-object identification. In 2022 researchers patched their way through GPT-2 small and found around twenty-six heads in seven roles: heads that notice a name is duplicated, heads that suppress the duplicated one, heads that copy the remaining name to the output. Removing the circuit removed the behaviour. Keeping only the circuit kept most of it. It took months."}]},{"t":"p","runs":[{"text":"Other circuits followed in small models: comparing years, completing docstrings, the induction behaviour from module two. Each is narrow, each is verified by intervention, and each is in a model far smaller than anything you use."}]},{"t":"h2","id":"automating-the-search","runs":[{"text":"Automating the search"}]},{"t":"p","runs":[{"text":"Doing the indirect-object circuit by hand was not going to scale. From 2023, automated methods prune the model’s full graph down to the edges that matter, using patching or its gradient approximation as the score. That cut circuit discovery in small models from months to hours."}]},{"t":"p","runs":[{"text":"In 2025 the approach was combined with the dictionaries of the previous lesson. Instead of heads and neurons, the nodes are dictionary features, and the method traces which features at one layer drive which at the next. Applied to a production-scale model, these attribution graphs gave the first mechanistic accounts of behaviours people care about: a model completing a two-step fact by passing through an intermediate concept, a model choosing a rhyme word several tokens before it reaches the end of the line and then writing toward it. The published caveat is that each graph explains a fraction of the behaviour, with the rest diffuse or in the dictionary’s dark matter."}]},{"t":"h2","id":"what-explained-means-here","runs":[{"text":"What “explained” means here"}]},{"t":"p","runs":[{"text":"A circuit is reported with a faithfulness figure. Eighty per cent means the circuit alone recovers eighty per cent of the difference in the answer’s logit; the remaining twenty is spread across components too small to name. There is no circuit for “answering questions” or “being helpful”. There are circuits for behaviours narrow enough to construct a clean-versus-corrupted pair for, and a mechanistic account of a model is a growing pile of those, not a single diagram."}]},{"t":"h2","id":"three-things-that-go-wrong","runs":[{"text":"Three things that go wrong"}]},{"t":"p","runs":[{"text":"The baseline decides the map.","b":true},{"text":" Patching measures the difference between two prompts. Choose a corrupted prompt that differs in several ways and the grid lights up for all of them at once. The art is in constructing pairs that differ in exactly one thing."}]},{"t":"p","runs":[{"text":"Self-repair.","b":true},{"text":" Remove a head and, in many models, another head partly takes over its job on the same forward pass. Single ablations therefore understate how much a component matters, and a component that looks unimportant alone may be one of two that cover for each other."}]},{"t":"p","runs":[{"text":"Sufficiency is not the whole story.","b":true},{"text":" A cell that recovers the answer when patched shows the information is there and usable. It does not show the model took that route unpatched. Sufficiency and necessity are both needed, and both are usually reported."}]},{"t":"h2","id":"what-you-can-do-with-it","runs":[{"text":"What you can do with it"}]},{"t":"p","runs":[{"text":"Patching is the only method in this module that yields a "},{"text":"causal","b":true},{"text":" claim, and that makes it the ground truth against which probes and dictionaries are checked. In practice it is how you locate a behaviour before trying to change it, which is what the next two lessons do, first by pushing on a direction and then by editing a weight, and it is the reason you can trust the phrase “this is where the fact lives” when it appears."}]}],"check":{"question":[{"text":"Patching the clean activation into the corrupted run at layer 15, last-token position, restores 85 per cent of the correct answer’s logit advantage. What has been shown?"}],"options":[{"text":[{"text":"That layer 15’s weights store the fact, so editing those weights would change what the model believes about it."}],"correct":false,"tempting":[{"text":"Confuses an activation (a value computed for this prompt) with the weights that computed it; where the fact is stored is a separate question the later editing lesson addresses."}]},{"text":[{"text":"That the information needed for the answer is present at that layer and position and is causally sufficient there to recover most of the behaviour."}],"correct":true,"tempting":null},{"text":[{"text":"That the layers after 15 contribute nothing to this answer, since the correct information is already fully present by then."}],"correct":false,"tempting":[{"text":"Later layers still have to read the patched information and turn it into the output; the patch works because they do that work, not because they are idle."}]},{"text":[{"text":"That a linear probe at layer 15 would decode the answer with roughly 85 per cent accuracy, because patching and probing measure the same underlying quantity in different units."}],"correct":false,"tempting":[{"text":"They measure different things (decodability versus causal effect) and a property can be highly decodable at a layer the model does not use, or used at a layer where a linear probe reads it poorly."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"sparse-autoencoders","title":"Sparse autoencoders: a dictionary for the residual stream","minutes":10},"next":{"slug":"steering-with-directions","title":"Steering: pushing on a direction","minutes":9}}}