{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"where-facts-live","title":"Where a fact lives, and whether you can edit it","url":"https://addaly.com/learn/how-llms-work/where-facts-live","position":74,"of":78,"minutes":9,"answer":"Causal tracing shows a fact is retrieved by mid-layer feed-forward blocks at the subject's position and copied forward by attention, and editing that lookup works on its own terms while failing on the reverse question.","keyIdea":"Causal tracing shows a fact is retrieved by mid-layer feed-forward blocks at the subject's position and copied forward by attention, and editing that lookup works on its own terms while failing on the reverse question, on neighbours, and after a few dozen edits."},"body":{"format":"markdown","text":"## The experiment\n\nTake \"The Eiffel Tower is in the city of\" and corrupt the subject: add noise to the token embeddings of \"Eiffel Tower\" so the model no longer knows what it is being asked about. The answer degrades. Now restore hidden states from the clean run, one at a time, at each layer and position (the patching of two lessons ago, first done at scale in exactly this experiment in 2022) and watch which single restoration brings \"Paris\" back.\n\nThe result is consistent across models. The strongest recovery comes from the **middle layers**, at the **last token of the subject**, and specifically from the feed-forward blocks there. Restore the mid-layer feed-forward output at the \"Tower\" position and the model recovers the answer; restore attention at the same place and it mostly does not.\n\nThis is module two's key-value picture, now with evidence. The representation of the subject, assembled by the time it reaches the middle of the stack, acts as a key; the feed-forward layers at that position return the subject's attributes as the value. Then, in later layers, attention **moves** the retrieved attribute from the subject's position to the final position, where the unembedding reads it out. Lookup, then copy. That two-stage image has held up under patching in several model families and is the concrete thing to carry away.\n\n## Editing\n\nIf a fact is a key-to-value lookup in a particular matrix, you can change the value. The method that followed the tracing does exactly that: a rank-one update to one mid-layer feed-forward weight matrix, chosen so that the key for \"Eiffel Tower\" now returns \"Rome\". After the edit the model says the Eiffel Tower is in Rome and, in the good cases, says so consistently across paraphrases: asked where to fly to see it, it answers Rome. A later method spreads thousands of such edits across several layers at once.\n\nIt works, on its own terms, and it is a beautiful confirmation that the tracing found something real. Then it breaks, in five ways you can now predict.\n\n## Why editing breaks\n\n**The reversal curse.** The edit rewrote what the key \"Eiffel Tower\" returns. Ask \"what famous tower is in Rome?\" and the model still says the Colosseum; ask \"what is in Paris?\" and it still mentions the Eiffel Tower. One key was changed, and module six explained that the reverse direction is a different key.\n\n**Bleeding.** Superposition means nearby keys share directions. After the edit, \"Tower Bridge is in\" and \"the Louvre is in\" sometimes drift toward Rome. An edit sharp enough not to bleed often fails to generalise to paraphrases; one broad enough to generalise bleeds. The two are the same dial.\n\n**Sequential collapse.** Apply a few dozen single edits one after another and the model's general ability degrades: perplexity rises, unrelated facts flip, fluency suffers. The batched method is more robust and not immune. Editing is a series of small wounds.\n\n**Localisation and editability come apart.** A 2023 result showed that the layers where tracing localises a fact are not reliably the best layers to edit it; edits at other layers work about as well. So \"where it lives\" and \"where to change it\" are different questions, which is uncomfortable for the picture above and honestly unresolved.\n\n**It edits the lookup, not the reasoning.** A model edited to place the tower in Rome will still, in a long answer, reason from Parisian context it retrieves through other keys. Facts are stored redundantly across many associations and one edit touches one of them.\n\n## What it is for\n\nEditing is a research instrument for studying how facts are represented, and it is a very good one. It is not a maintenance tool. Nobody updates a production model's knowledge by editing weights, and no vendor who claims to should be believed without the reverse-question test. When a fact changes, the answer is retrieval: module four, and module six's lesson on weights versus context. When a fact must be *forgotten*, the answer is the next lesson, and it is mostly that it cannot be done cleanly.\n\n## Try it once\n\nThe reference implementations, and a library that wraps several editing methods, run on a 1- or 2-billion-parameter model in a free notebook. Edit the capital of France. Ask the reverse question. Ask about the Louvre. Apply twenty more edits and re-run a general benchmark. Half an hour of this does more to calibrate your reading of \"we updated the model's knowledge\" than any amount of argument, and it leaves you with the durable part: a fact is a lookup at the subject's position in the middle layers, copied forward by attention: findable, fragile, and one of many."},"check":{"question":"After a rank-one weight edit makes a model say the Eiffel Tower is in Rome, it still answers \"What famous tower is in Paris?\" with \"the Eiffel Tower\". Why?","options":[{"text":"The edit was applied at the wrong layer; tracing the reverse question would have located a later layer where both directions are stored together.","correct":false,"tempting":"Localisation and editability do come apart, but no layer stores both directions together; the reverse association is a different key wherever it is."},{"text":"The edit only raised the output layer's bias toward the token Rome, so it appears whenever the tower is mentioned but nothing about Paris was changed.","correct":false,"tempting":"A rank-one edit to a mid-layer feed-forward matrix is not an output bias; if it were, Rome would leak into unrelated answers far more than it does."},{"text":"The edit rewrote one key-to-value lookup for the subject, and the reverse question uses a different key.","correct":true},{"text":"The reverse question retrieves from the attention layers, which store facts redundantly in a form that weight edits to the feed-forward layers cannot reach, so the model holds two copies and only one was changed.","correct":false,"tempting":"Facts are stored redundantly, but across many feed-forward keys, not in a separate attention copy; attention moves retrieved attributes rather than storing them."}]},"prev":{"slug":"steering-with-directions","title":"Steering: pushing on a direction","url":"https://addaly.com/learn/how-llms-work/steering-with-directions"},"next":{"slug":"is-the-reasoning-faithful","title":"Is the written reasoning what the model did?","url":"https://addaly.com/learn/how-llms-work/is-the-reasoning-faithful"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where a fact lives, and whether you can edit it', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/where-facts-live (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where a fact lives, and whether you can edit it","courseTitle":"How a Language Model Actually Works","module":{"index":8,"slug":"looking-inside","title":"Looking inside: what the weights actually hold"},"answer":null,"keyIdea":"Causal tracing shows a fact is retrieved by mid-layer feed-forward blocks at the subject’s position and copied forward by attention, and editing that lookup works on its own terms while failing on the reverse question, on neighbours, and after a few dozen edits.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-experiment","runs":[{"text":"The experiment"}]},{"t":"p","runs":[{"text":"Take “The Eiffel Tower is in the city of” and corrupt the subject: add noise to the token embeddings of “Eiffel Tower” so the model no longer knows what it is being asked about. The answer degrades. Now restore hidden states from the clean run, one at a time, at each layer and position (the patching of two lessons ago, first done at scale in exactly this experiment in 2022) and watch which single restoration brings “Paris” back."}]},{"t":"p","runs":[{"text":"The result is consistent across models. The strongest recovery comes from the "},{"text":"middle layers","b":true},{"text":", at the "},{"text":"last token of the subject","b":true},{"text":", and specifically from the feed-forward blocks there. Restore the mid-layer feed-forward output at the “Tower” position and the model recovers the answer; restore attention at the same place and it mostly does not."}]},{"t":"p","runs":[{"text":"This is module two’s key-value picture, now with evidence. The representation of the subject, assembled by the time it reaches the middle of the stack, acts as a key; the feed-forward layers at that position return the subject’s attributes as the value. Then, in later layers, attention "},{"text":"moves","b":true},{"text":" the retrieved attribute from the subject’s position to the final position, where the unembedding reads it out. Lookup, then copy. That two-stage image has held up under patching in several model families and is the concrete thing to carry away."}]},{"t":"h2","id":"editing","runs":[{"text":"Editing"}]},{"t":"p","runs":[{"text":"If a fact is a key-to-value lookup in a particular matrix, you can change the value. The method that followed the tracing does exactly that: a rank-one update to one mid-layer feed-forward weight matrix, chosen so that the key for “Eiffel Tower” now returns “Rome”. After the edit the model says the Eiffel Tower is in Rome and, in the good cases, says so consistently across paraphrases: asked where to fly to see it, it answers Rome. A later method spreads thousands of such edits across several layers at once."}]},{"t":"p","runs":[{"text":"It works, on its own terms, and it is a beautiful confirmation that the tracing found something real. Then it breaks, in five ways you can now predict."}]},{"t":"h2","id":"why-editing-breaks","runs":[{"text":"Why editing breaks"}]},{"t":"p","runs":[{"text":"The reversal curse.","b":true},{"text":" The edit rewrote what the key “Eiffel Tower” returns. Ask “what famous tower is in Rome?” and the model still says the Colosseum; ask “what is in Paris?” and it still mentions the Eiffel Tower. One key was changed, and module six explained that the reverse direction is a different key."}]},{"t":"p","runs":[{"text":"Bleeding.","b":true},{"text":" Superposition means nearby keys share directions. After the edit, “Tower Bridge is in” and “the Louvre is in” sometimes drift toward Rome. An edit sharp enough not to bleed often fails to generalise to paraphrases; one broad enough to generalise bleeds. The two are the same dial."}]},{"t":"p","runs":[{"text":"Sequential collapse.","b":true},{"text":" Apply a few dozen single edits one after another and the model’s general ability degrades: perplexity rises, unrelated facts flip, fluency suffers. The batched method is more robust and not immune. Editing is a series of small wounds."}]},{"t":"p","runs":[{"text":"Localisation and editability come apart.","b":true},{"text":" A 2023 result showed that the layers where tracing localises a fact are not reliably the best layers to edit it; edits at other layers work about as well. So “where it lives” and “where to change it” are different questions, which is uncomfortable for the picture above and honestly unresolved."}]},{"t":"p","runs":[{"text":"It edits the lookup, not the reasoning.","b":true},{"text":" A model edited to place the tower in Rome will still, in a long answer, reason from Parisian context it retrieves through other keys. Facts are stored redundantly across many associations and one edit touches one of them."}]},{"t":"h2","id":"what-it-is-for","runs":[{"text":"What it is for"}]},{"t":"p","runs":[{"text":"Editing is a research instrument for studying how facts are represented, and it is a very good one. It is not a maintenance tool. Nobody updates a production model’s knowledge by editing weights, and no vendor who claims to should be believed without the reverse-question test. When a fact changes, the answer is retrieval: module four, and module six’s lesson on weights versus context. When a fact must be "},{"text":"forgotten","i":true},{"text":", the answer is the next lesson, and it is mostly that it cannot be done cleanly."}]},{"t":"h2","id":"try-it-once","runs":[{"text":"Try it once"}],"role":"try"},{"t":"p","runs":[{"text":"The reference implementations, and a library that wraps several editing methods, run on a 1- or 2-billion-parameter model in a free notebook. Edit the capital of France. Ask the reverse question. Ask about the Louvre. Apply twenty more edits and re-run a general benchmark. Half an hour of this does more to calibrate your reading of “we updated the model’s knowledge” than any amount of argument, and it leaves you with the durable part: a fact is a lookup at the subject’s position in the middle layers, copied forward by attention: findable, fragile, and one of many."}]}],"check":{"question":[{"text":"After a rank-one weight edit makes a model say the Eiffel Tower is in Rome, it still answers “What famous tower is in Paris?” with “the Eiffel Tower”. Why?"}],"options":[{"text":[{"text":"The edit was applied at the wrong layer; tracing the reverse question would have located a later layer where both directions are stored together."}],"correct":false,"tempting":[{"text":"Localisation and editability do come apart, but no layer stores both directions together; the reverse association is a different key wherever it is."}]},{"text":[{"text":"The edit only raised the output layer’s bias toward the token Rome, so it appears whenever the tower is mentioned but nothing about Paris was changed."}],"correct":false,"tempting":[{"text":"A rank-one edit to a mid-layer feed-forward matrix is not an output bias; if it were, Rome would leak into unrelated answers far more than it does."}]},{"text":[{"text":"The edit rewrote one key-to-value lookup for the subject, and the reverse question uses a different key."}],"correct":true,"tempting":null},{"text":[{"text":"The reverse question retrieves from the attention layers, which store facts redundantly in a form that weight edits to the feed-forward layers cannot reach, so the model holds two copies and only one was changed."}],"correct":false,"tempting":[{"text":"Facts are stored redundantly, but across many feed-forward keys, not in a separate attention copy; attention moves retrieved attributes rather than storing them."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"steering-with-directions","title":"Steering: pushing on a direction","minutes":9},"next":{"slug":"is-the-reasoning-faithful","title":"Is the written reasoning what the model did?","minutes":9}}}