Where a fact lives, and whether you can edit it

The experiment

Take “The Eiffel Tower is in the city of” and corrupt the subject: add noise to the token embeddings of “Eiffel Tower” so the model no longer knows what it is being asked about. The answer degrades. Now restore hidden states from the clean run, one at a time, at each layer and position (the patching of two lessons ago, first done at scale in exactly this experiment in 2022) and watch which single restoration brings “Paris” back.

The result is consistent across models. The strongest recovery comes from the middle layers, at the last token of the subject, and specifically from the feed-forward blocks there. Restore the mid-layer feed-forward output at the “Tower” position and the model recovers the answer; restore attention at the same place and it mostly does not.

This is module two’s key-value picture, now with evidence. The representation of the subject, assembled by the time it reaches the middle of the stack, acts as a key; the feed-forward layers at that position return the subject’s attributes as the value. Then, in later layers, attention moves the retrieved attribute from the subject’s position to the final position, where the unembedding reads it out. Lookup, then copy. That two-stage image has held up under patching in several model families and is the concrete thing to carry away.

Editing

If a fact is a key-to-value lookup in a particular matrix, you can change the value. The method that followed the tracing does exactly that: a rank-one update to one mid-layer feed-forward weight matrix, chosen so that the key for “Eiffel Tower” now returns “Rome”. After the edit the model says the Eiffel Tower is in Rome and, in the good cases, says so consistently across paraphrases: asked where to fly to see it, it answers Rome. A later method spreads thousands of such edits across several layers at once.

It works, on its own terms, and it is a beautiful confirmation that the tracing found something real. Then it breaks, in five ways you can now predict.

Why editing breaks

The reversal curse. The edit rewrote what the key “Eiffel Tower” returns. Ask “what famous tower is in Rome?” and the model still says the Colosseum; ask “what is in Paris?” and it still mentions the Eiffel Tower. One key was changed, and module six explained that the reverse direction is a different key.

Bleeding. Superposition means nearby keys share directions. After the edit, “Tower Bridge is in” and “the Louvre is in” sometimes drift toward Rome. An edit sharp enough not to bleed often fails to generalise to paraphrases; one broad enough to generalise bleeds. The two are the same dial.

Sequential collapse. Apply a few dozen single edits one after another and the model’s general ability degrades: perplexity rises, unrelated facts flip, fluency suffers. The batched method is more robust and not immune. Editing is a series of small wounds.

Localisation and editability come apart. A 2023 result showed that the layers where tracing localises a fact are not reliably the best layers to edit it; edits at other layers work about as well. So “where it lives” and “where to change it” are different questions, which is uncomfortable for the picture above and honestly unresolved.

It edits the lookup, not the reasoning. A model edited to place the tower in Rome will still, in a long answer, reason from Parisian context it retrieves through other keys. Facts are stored redundantly across many associations and one edit touches one of them.

What it is for

Editing is a research instrument for studying how facts are represented, and it is a very good one. It is not a maintenance tool. Nobody updates a production model’s knowledge by editing weights, and no vendor who claims to should be believed without the reverse-question test. When a fact changes, the answer is retrieval: module four, and module six’s lesson on weights versus context. When a fact must be forgotten, the answer is the next lesson, and it is mostly that it cannot be done cleanly.

Try it once

The reference implementations, and a library that wraps several editing methods, run on a 1- or 2-billion-parameter model in a free notebook. Edit the capital of France. Ask the reverse question. Ask about the Louvre. Apply twenty more edits and re-run a general benchmark. Half an hour of this does more to calibrate your reading of “we updated the model’s knowledge” than any amount of argument, and it leaves you with the durable part: a fact is a lookup at the subject’s position in the middle layers, copied forward by attention: findable, fragile, and one of many.