{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"knowledge-in-weights-versus-context","title":"Two places a fact can live, and what happens when they disagree","url":"https://addaly.com/learn/how-llms-work/knowledge-in-weights-versus-context","position":52,"of":78,"minutes":9,"answer":"A memorised fact and a fact in the context reach the output as two contributions added into the same vector.","keyIdea":"A memorised fact and a fact in the context reach the output as two contributions added into the same vector, so the larger one wins and nothing in the architecture prefers what the model was told over what it remembers."},"body":{"format":"markdown","text":"## Two places a fact can live\n\nA fact reaches the output by one of two routes. It was in the pretraining data and is now encoded in the weights: the feed-forward key-value store of module two. Or it is in the context window right now, put there by you, a document, or a retrieval step, and reaches the output through attention.\n\nModule four said the context is not memory. Module five said the weights are where knowledge comes from. What neither said is what happens when the two disagree, and that case is where retrieval systems fail.\n\n## The experiment\n\nTake a fact the model knows well: the chief executive of a large company, the year a treaty was signed, the price of a well-documented product. Write a short passage stating a different value. Put the passage in the prompt and ask the question.\n\nStudies of this setup, from 2023 onward, find a consistent pattern. When the substituted fact is **coherent** (a plausible person's name, a nearby year) most models follow the passage most of the time. When it is **implausible** (a capital city in the wrong country) models revert to what they memorised far more often. With several passages that disagree, models tend to prefer the one that matches their prior, or the majority, or whichever came last, in proportions that vary by model. Instruction-tuned and larger models follow the context more reliably than base models, and none does so always.\n\nThe rate at which a model overrides its own memory with your document has a name in the literature (the context-override or contextual-faithfulness rate) and it is a property you should measure for any model you deploy with retrieval.\n\n## Why, in the mechanism you already have\n\nBoth routes produce contributions to the same final vector. Module two showed that every block **adds** to a running total: attention adds what it copied from the context, the feed-forward layer adds what it looked up from the weights, and the logits are read off the sum.\n\nA fact seen ten thousand times in pretraining produces a large, sharp contribution. A single sentence in a document, especially one far from the question, produces a smaller one. When they point at different tokens, the larger contribution wins the logit, and nothing in the architecture ranks \"what I was told\" above \"what I remember\". It is addition. Whichever is bigger, is bigger.\n\nThis is also why implausible substitutions lose more often. The memorised fact does not just contribute the answer; it contributes an entire neighbourhood of associations (the country, the language, the currency) that all agree with each other and all disagree with the passage.\n\n> Figure: Your document says one thing, the weights say another. A two by two grid, rows Substitution is plausible and Substitution is implausible, columns Fact seen often in training and Fact seen rarely: Substitution is plausible and Fact seen often in training: often follows you; Substitution is plausible and Fact seen rarely: usually follows you; Substitution is implausible and Fact seen often in training: often reverts; Substitution is implausible and Fact seen rarely: usually follows you. Both routes add into the same vector and the larger contribution wins the logit. Nothing in the architecture ranks what the model was told above what it remembers, which is why the override rate is a number to measure for your model rather than a setting to switch on.\n\n## Three consequences\n\n**Your data gets \"corrected\".** A product catalogue with a new price, a policy document with a changed limit, an internal name for something the public calls something else. The model gives the old value it memorised, with full confidence, in a system whose whole purpose was to give the new one.\n\n**Gaps get filled from memory.** When the document is silent on a detail, the model supplies one from the weights. This is hallucination wearing the costume of reading: the answer cites the document and contains a fact the document never stated.\n\n**The model cannot tell which is newer.** It has no timestamps on its own knowledge. A retrieved passage from this morning and a memory from three years ago are two contributions of different sizes.\n\n## Two tests, ten minutes each\n\n**The counterfactual test.** Take twenty retrieved documents and change one fact in each: a number, a name, a date. Ask the question. Count how often the answer follows your change. If the count is low, your retrieval layer is decorative: the model was answering from memory all along and the documents were reassurance.\n\n**The silent-document test.** Remove the answer from the document entirely. Ask the same question. Count how often the model says the document does not contain it. The complement of that number is the rate at which your system invents answers under the cover of a citation.\n\nBoth belong in a permanent evaluation set, re-run on every model change, because the override rate moves between versions.\n\n## Mitigations, and what they do\n\n- **Put the passage next to the question.** Module four's position bias applies: a fact in the middle of a long prompt contributes less than the same fact just above the query.\n- **Quote, then answer.** Ask the model to copy the supporting sentence verbatim before answering. The copy puts the fact into the most recent positions, in the model's own output, which strengthens the contextual route. This is one of the few prompt-level interventions with a mechanical reason to work.\n- **Say what to do when the document is silent.** \"If the passage does not contain the answer, say so.\" Reduces the gap-filling rate; does not eliminate it.\n- **Do not reach for fine-tuning.** Module five was clear that fine-tuning changes form rather than facts. Training a model to \"prefer the context\" is itself a learned behaviour that competes with the memorised prior on the same additive terms, and it can be measured to help; it cannot be relied on to hold.\n\n## What is not controllable\n\nThere is no setting for \"trust the document\". The balance between the two routes is a fact about the weights and the prompt, adjustable at the margin and never fixed. A system that must give the current value of something that changes should get that value from a tool call the model cannot override, and use the model to phrase it, which is the division of labour module four's tool lesson described, applied to the one case where it matters most."},"check":{"question":"A retrieval system's fetched document says a product costs ₹4,999. The model answers ₹3,999, which was the price when its training data was collected. What is the mechanism?","options":[{"text":"The retriever returned a chunk from an older version of the document, so the model was correctly reporting exactly what it had been given.","correct":false,"tempting":"The most common first guess, and worth ruling out, but the question states the fetched document carried the new price, so retrieval was not the failure."},{"text":"The context window was too small for the document, so the price line was truncated before the model saw it.","correct":false,"tempting":"Truncation does silently drop content, but a single price line in a fetched document is exactly the kind of short context that fits; the answer being the old price points at memory, not absence."},{"text":"The memorised price and the document's price push on the same logits, and the stronger memorised push won.","correct":true},{"text":"Models treat numbers inside quoted documents as illustrative examples rather than facts, and only trust numbers stated in the user's own message.","correct":false,"tempting":"Invents a rule the architecture does not have; the model makes no such distinction between sources, which is precisely the problem."}]},"prev":{"slug":"sycophancy","title":"Sycophancy: agreement as a trained reflex","url":"https://addaly.com/learn/how-llms-work/sycophancy"},"next":{"slug":"the-reversal-curse","title":"What it means to know a fact: the reversal curse","url":"https://addaly.com/learn/how-llms-work/the-reversal-curse"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Two places a fact can live, and what happens when they disagree', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/knowledge-in-weights-versus-context (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Two places a fact can live, and what happens when they disagree","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"A memorised fact and a fact in the context reach the output as two contributions added into the same vector, so the larger one wins and nothing in the architecture prefers what the model was told over what it remembers.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Fact seen often in training","Fact seen rarely"],"rows":["Substitution is plausible","Substitution is implausible"],"type":"matrix","cells":[["often follows you","usually follows you"],["often reverts","usually follows you"]],"title":"Your document says one thing, the weights say another","caption":"Both routes add into the same vector and the larger contribution wins the logit. Nothing in the architecture ranks what the model was told above what it remembers, which is why the override rate is a number to measure for your model rather than a setting to switch on.","cellLabels":[["A different but believable chief executive","A niche detail the weights barely hold"],["A capital city moved into the wrong country","Little memorised contribution to compete with"]]},"alt":"Figure: Your document says one thing, the weights say another. A two by two grid, rows Substitution is plausible and Substitution is implausible, columns Fact seen often in training and Fact seen rarely: Substitution is plausible and Fact seen often in training: often follows you; Substitution is plausible and Fact seen rarely: usually follows you; Substitution is implausible and Fact seen often in training: often reverts; Substitution is implausible and Fact seen rarely: usually follows you. Both routes add into the same vector and the larger contribution wins the logit. Nothing in the architecture ranks what the model was told above what it remembers, which is why the override rate is a number to measure for your model rather than a setting to switch on."}},"blocks":[{"t":"h2","id":"two-places-a-fact-can-live","runs":[{"text":"Two places a fact can live"}]},{"t":"p","runs":[{"text":"A fact reaches the output by one of two routes. It was in the pretraining data and is now encoded in the weights: the feed-forward key-value store of module two. Or it is in the context window right now, put there by you, a document, or a retrieval step, and reaches the output through attention."}]},{"t":"p","runs":[{"text":"Module four said the context is not memory. Module five said the weights are where knowledge comes from. What neither said is what happens when the two disagree, and that case is where retrieval systems fail."}]},{"t":"h2","id":"the-experiment","runs":[{"text":"The experiment"}]},{"t":"p","runs":[{"text":"Take a fact the model knows well: the chief executive of a large company, the year a treaty was signed, the price of a well-documented product. Write a short passage stating a different value. Put the passage in the prompt and ask the question."}]},{"t":"p","runs":[{"text":"Studies of this setup, from 2023 onward, find a consistent pattern. When the substituted fact is "},{"text":"coherent","b":true},{"text":" (a plausible person’s name, a nearby year) most models follow the passage most of the time. When it is "},{"text":"implausible","b":true},{"text":" (a capital city in the wrong country) models revert to what they memorised far more often. With several passages that disagree, models tend to prefer the one that matches their prior, or the majority, or whichever came last, in proportions that vary by model. Instruction-tuned and larger models follow the context more reliably than base models, and none does so always."}]},{"t":"p","runs":[{"text":"The rate at which a model overrides its own memory with your document has a name in the literature (the context-override or contextual-faithfulness rate) and it is a property you should measure for any model you deploy with retrieval."}]},{"t":"h2","id":"why-in-the-mechanism-you-already-have","runs":[{"text":"Why, in the mechanism you already have"}]},{"t":"p","runs":[{"text":"Both routes produce contributions to the same final vector. Module two showed that every block "},{"text":"adds","b":true},{"text":" to a running total: attention adds what it copied from the context, the feed-forward layer adds what it looked up from the weights, and the logits are read off the sum."}]},{"t":"p","runs":[{"text":"A fact seen ten thousand times in pretraining produces a large, sharp contribution. A single sentence in a document, especially one far from the question, produces a smaller one. When they point at different tokens, the larger contribution wins the logit, and nothing in the architecture ranks “what I was told” above “what I remember”. It is addition. Whichever is bigger, is bigger."}]},{"t":"p","runs":[{"text":"This is also why implausible substitutions lose more often. The memorised fact does not just contribute the answer; it contributes an entire neighbourhood of associations (the country, the language, the currency) that all agree with each other and all disagree with the passage."}]},{"t":"h2","id":"three-consequences","runs":[{"text":"Three consequences"}]},{"t":"p","runs":[{"text":"Your data gets “corrected”.","b":true},{"text":" A product catalogue with a new price, a policy document with a changed limit, an internal name for something the public calls something else. The model gives the old value it memorised, with full confidence, in a system whose whole purpose was to give the new one."}]},{"t":"p","runs":[{"text":"Gaps get filled from memory.","b":true},{"text":" When the document is silent on a detail, the model supplies one from the weights. This is hallucination wearing the costume of reading: the answer cites the document and contains a fact the document never stated."}]},{"t":"p","runs":[{"text":"The model cannot tell which is newer.","b":true},{"text":" It has no timestamps on its own knowledge. A retrieved passage from this morning and a memory from three years ago are two contributions of different sizes."}]},{"t":"h2","id":"two-tests-ten-minutes-each","runs":[{"text":"Two tests, ten minutes each"}]},{"t":"p","runs":[{"text":"The counterfactual test.","b":true},{"text":" Take twenty retrieved documents and change one fact in each: a number, a name, a date. Ask the question. Count how often the answer follows your change. If the count is low, your retrieval layer is decorative: the model was answering from memory all along and the documents were reassurance."}]},{"t":"p","runs":[{"text":"The silent-document test.","b":true},{"text":" Remove the answer from the document entirely. Ask the same question. Count how often the model says the document does not contain it. The complement of that number is the rate at which your system invents answers under the cover of a citation."}]},{"t":"p","runs":[{"text":"Both belong in a permanent evaluation set, re-run on every model change, because the override rate moves between versions."}]},{"t":"h2","id":"mitigations-and-what-they-do","runs":[{"text":"Mitigations, and what they do"}]},{"t":"ul","items":[[{"text":"Put the passage next to the question.","b":true},{"text":" Module four’s position bias applies: a fact in the middle of a long prompt contributes less than the same fact just above the query."}],[{"text":"Quote, then answer.","b":true},{"text":" Ask the model to copy the supporting sentence verbatim before answering. The copy puts the fact into the most recent positions, in the model’s own output, which strengthens the contextual route. This is one of the few prompt-level interventions with a mechanical reason to work."}],[{"text":"Say what to do when the document is silent.","b":true},{"text":" “If the passage does not contain the answer, say so.” Reduces the gap-filling rate; does not eliminate it."}],[{"text":"Do not reach for fine-tuning.","b":true},{"text":" Module five was clear that fine-tuning changes form rather than facts. Training a model to “prefer the context” is itself a learned behaviour that competes with the memorised prior on the same additive terms, and it can be measured to help; it cannot be relied on to hold."}]]},{"t":"h2","id":"what-is-not-controllable","runs":[{"text":"What is not controllable"}]},{"t":"p","runs":[{"text":"There is no setting for “trust the document”. The balance between the two routes is a fact about the weights and the prompt, adjustable at the margin and never fixed. A system that must give the current value of something that changes should get that value from a tool call the model cannot override, and use the model to phrase it, which is the division of labour module four’s tool lesson described, applied to the one case where it matters most."}]}],"check":{"question":[{"text":"A retrieval system’s fetched document says a product costs ₹4,999. The model answers ₹3,999, which was the price when its training data was collected. What is the mechanism?"}],"options":[{"text":[{"text":"The retriever returned a chunk from an older version of the document, so the model was correctly reporting exactly what it had been given."}],"correct":false,"tempting":[{"text":"The most common first guess, and worth ruling out, but the question states the fetched document carried the new price, so retrieval was not the failure."}]},{"text":[{"text":"The context window was too small for the document, so the price line was truncated before the model saw it."}],"correct":false,"tempting":[{"text":"Truncation does silently drop content, but a single price line in a fetched document is exactly the kind of short context that fits; the answer being the old price points at memory, not absence."}]},{"text":[{"text":"The memorised price and the document’s price push on the same logits, and the stronger memorised push won."}],"correct":true,"tempting":null},{"text":[{"text":"Models treat numbers inside quoted documents as illustrative examples rather than facts, and only trust numbers stated in the user’s own message."}],"correct":false,"tempting":[{"text":"Invents a rule the architecture does not have; the model makes no such distinction between sources, which is precisely the problem."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"sycophancy","title":"Sycophancy: agreement as a trained reflex","minutes":8},"next":{"slug":"the-reversal-curse","title":"What it means to know a fact: the reversal curse","minutes":8}}}