Two places a fact can live, and what happens when they disagree

Two places a fact can live

A fact reaches the output by one of two routes. It was in the pretraining data and is now encoded in the weights: the feed-forward key-value store of module two. Or it is in the context window right now, put there by you, a document, or a retrieval step, and reaches the output through attention.

Module four said the context is not memory. Module five said the weights are where knowledge comes from. What neither said is what happens when the two disagree, and that case is where retrieval systems fail.

The experiment

Take a fact the model knows well: the chief executive of a large company, the year a treaty was signed, the price of a well-documented product. Write a short passage stating a different value. Put the passage in the prompt and ask the question.

Studies of this setup, from 2023 onward, find a consistent pattern. When the substituted fact is coherent (a plausible person’s name, a nearby year) most models follow the passage most of the time. When it is implausible (a capital city in the wrong country) models revert to what they memorised far more often. With several passages that disagree, models tend to prefer the one that matches their prior, or the majority, or whichever came last, in proportions that vary by model. Instruction-tuned and larger models follow the context more reliably than base models, and none does so always.

The rate at which a model overrides its own memory with your document has a name in the literature (the context-override or contextual-faithfulness rate) and it is a property you should measure for any model you deploy with retrieval.

Why, in the mechanism you already have

Both routes produce contributions to the same final vector. Module two showed that every block adds to a running total: attention adds what it copied from the context, the feed-forward layer adds what it looked up from the weights, and the logits are read off the sum.

A fact seen ten thousand times in pretraining produces a large, sharp contribution. A single sentence in a document, especially one far from the question, produces a smaller one. When they point at different tokens, the larger contribution wins the logit, and nothing in the architecture ranks “what I was told” above “what I remember”. It is addition. Whichever is bigger, is bigger.

This is also why implausible substitutions lose more often. The memorised fact does not just contribute the answer; it contributes an entire neighbourhood of associations (the country, the language, the currency) that all agree with each other and all disagree with the passage.

Three consequences

Your data gets “corrected”. A product catalogue with a new price, a policy document with a changed limit, an internal name for something the public calls something else. The model gives the old value it memorised, with full confidence, in a system whose whole purpose was to give the new one.

Gaps get filled from memory. When the document is silent on a detail, the model supplies one from the weights. This is hallucination wearing the costume of reading: the answer cites the document and contains a fact the document never stated.

The model cannot tell which is newer. It has no timestamps on its own knowledge. A retrieved passage from this morning and a memory from three years ago are two contributions of different sizes.

Two tests, ten minutes each

The counterfactual test. Take twenty retrieved documents and change one fact in each: a number, a name, a date. Ask the question. Count how often the answer follows your change. If the count is low, your retrieval layer is decorative: the model was answering from memory all along and the documents were reassurance.

The silent-document test. Remove the answer from the document entirely. Ask the same question. Count how often the model says the document does not contain it. The complement of that number is the rate at which your system invents answers under the cover of a citation.

Both belong in a permanent evaluation set, re-run on every model change, because the override rate moves between versions.

Mitigations, and what they do

• Put the passage next to the question. Module four’s position bias applies: a fact in the middle of a long prompt contributes less than the same fact just above the query.
• Quote, then answer. Ask the model to copy the supporting sentence verbatim before answering. The copy puts the fact into the most recent positions, in the model’s own output, which strengthens the contextual route. This is one of the few prompt-level interventions with a mechanical reason to work.
• Say what to do when the document is silent. “If the passage does not contain the answer, say so.” Reduces the gap-filling rate; does not eliminate it.
• Do not reach for fine-tuning. Module five was clear that fine-tuning changes form rather than facts. Training a model to “prefer the context” is itself a learned behaviour that competes with the memorised prior on the same additive terms, and it can be measured to help; it cannot be relied on to hold.

What is not controllable

There is no setting for “trust the document”. The balance between the two routes is a fact about the weights and the prompt, adjustable at the margin and never fixed. A system that must give the current value of something that changes should get that value from a tool call the model cannot override, and use the model to phrase it, which is the division of labour module four’s tool lesson described, applied to the one case where it matters most.