Patching: finding which parts do the work

The question the first three lessons cannot answer

A probe says a property is readable. A dictionary says a direction exists and what it fires on. Neither says whether the model uses it to produce the output in front of you, or where. For that you need to intervene, and the standard intervention is called activation patching.

The method

Choose two prompts that differ in one thing and produce different answers. Clean: “The Eiffel Tower is in the city of”: answer Paris. Corrupted: “The Colosseum is in the city of”: answer Rome.

Run both, and store every activation from the clean run. Now run the corrupted prompt again, but at one layer and one token position, overwrite the activation with the stored clean one. Measure how far the output moves back toward Paris.

Repeat for every layer and every position, and you get a grid: where in the network does the clean information, inserted alone, recover the clean answer? Bright cells are places where the answer-relevant information is present and causally sufficient. Dark cells are places where it is not.

def patch(activation, hook, clean_cache, pos):
    activation[:, pos] = clean_cache[hook.name][:, pos]
    return activation
# run the corrupted prompt with this hook at (layer, pos), read the Paris logit

That is the whole method, and with TransformerLens on an open model it is about twenty lines. Variants patch individual attention heads or feed-forward blocks instead of whole layers; patch from corrupted into clean to test necessity rather than sufficiency; and approximate the whole grid at once with gradients (attribution patching) when there are thousands of components to test.

What a circuit is

A circuit is a subgraph of components (particular heads, particular feed-forward features, and the paths between them) that together implement one behaviour, verified by patching: keep only those components’ contributions and most of the behaviour survives; remove only them and most of it goes.

The example module two mentioned is the clearest. “When Mary and John went to the shop, John gave a drink to”: the answer is Mary, and the task is called indirect-object identification. In 2022 researchers patched their way through GPT-2 small and found around twenty-six heads in seven roles: heads that notice a name is duplicated, heads that suppress the duplicated one, heads that copy the remaining name to the output. Removing the circuit removed the behaviour. Keeping only the circuit kept most of it. It took months.

Other circuits followed in small models: comparing years, completing docstrings, the induction behaviour from module two. Each is narrow, each is verified by intervention, and each is in a model far smaller than anything you use.

Automating the search

Doing the indirect-object circuit by hand was not going to scale. From 2023, automated methods prune the model’s full graph down to the edges that matter, using patching or its gradient approximation as the score. That cut circuit discovery in small models from months to hours.

In 2025 the approach was combined with the dictionaries of the previous lesson. Instead of heads and neurons, the nodes are dictionary features, and the method traces which features at one layer drive which at the next. Applied to a production-scale model, these attribution graphs gave the first mechanistic accounts of behaviours people care about: a model completing a two-step fact by passing through an intermediate concept, a model choosing a rhyme word several tokens before it reaches the end of the line and then writing toward it. The published caveat is that each graph explains a fraction of the behaviour, with the rest diffuse or in the dictionary’s dark matter.

What “explained” means here

A circuit is reported with a faithfulness figure. Eighty per cent means the circuit alone recovers eighty per cent of the difference in the answer’s logit; the remaining twenty is spread across components too small to name. There is no circuit for “answering questions” or “being helpful”. There are circuits for behaviours narrow enough to construct a clean-versus-corrupted pair for, and a mechanistic account of a model is a growing pile of those, not a single diagram.

Three things that go wrong

The baseline decides the map. Patching measures the difference between two prompts. Choose a corrupted prompt that differs in several ways and the grid lights up for all of them at once. The art is in constructing pairs that differ in exactly one thing.

Self-repair. Remove a head and, in many models, another head partly takes over its job on the same forward pass. Single ablations therefore understate how much a component matters, and a component that looks unimportant alone may be one of two that cover for each other.

Sufficiency is not the whole story. A cell that recovers the answer when patched shows the information is there and usable. It does not show the model took that route unpatched. Sufficiency and necessity are both needed, and both are usually reported.

What you can do with it

Patching is the only method in this module that yields a causal claim, and that makes it the ground truth against which probes and dictionaries are checked. In practice it is how you locate a behaviour before trying to change it, which is what the next two lessons do, first by pushing on a direction and then by editing a weight, and it is the reason you can trust the phrase “this is where the fact lives” when it appears.