Reducing hallucination by mechanism, not by instruction

What “do not hallucinate” does

Almost nothing you can measure. The instruction asks the model to distinguish its supported outputs from its unsupported ones, and it has no access to that distinction. Every token is produced the same way, by the same process, whether or not the fact it completes is true. The model that writes a correct date and the model that writes a plausible wrong one are the same model doing the same thing. Asking it to stop is asking for a self-knowledge that is not there.

Temperature zero does not help either. It picks the most probable token, and the most probable token can be the confident wrong one. The sampling lesson in the first module explained why.

What does help is a set of mechanisms that change what the model is given, what it is allowed to produce, and what is checked afterwards. Here they are, roughly in order of effect.

Retrieval quality

The largest lever by a distance. A model that has the right passage in front of it invents far less than one that does not, because completing text from a source is a different task from completing it from memory. Every improvement in the retrieval module (recall, ranking, ingestion) lowers the hallucination rate before any other technique is applied. If the two-by-two from the evaluation module shows retrieval misses, start there.

Abstention below a score

The reranking lesson set a cutoff: below a certain reranker score, nothing relevant was found. Use it. When the best candidate scores below the cutoff, the answer is “I could not find this in the available documents”, not five weak chunks and a guess. This trades some answerable questions for far fewer invented answers, and the trade is measured on the evaluation set: count the questions refused that had answers, and the questions answered wrongly that would have been refused, at each cutoff. The abstention path is a feature, and the interface should make it read as one.

Claims that cite, and citations that are checked

Require every factual sentence to carry a chunk id. Then check each one. The cheap check is existence: the id was in the retrieved set. The stronger check is support: a second, small call, or a free entailment model, asked whether chunk X supports sentence Y. Sentences that fail are removed or marked. This is claim-level verification, and it is the mechanism that turns “grounded” from a hope into a property. It costs one small call per answer and catches the confident sentence that cites a real chunk which says something else.

Constrained output

Where the answer space is known, make it an enumeration. A plan name from a list of four cannot be a fifth plan. A date must parse. A price must match a price in the retrieved data. The structured-output lessons built this; here its purpose is to make certain inventions impossible rather than merely unlikely.

Quote first

The grounding lesson in the prompting module asked the model to quote the relevant passage before answering. The mechanism is attention: producing the quote anchors the answer to text that exists. It is cheap and it measurably reduces unsupported claims, and it combines with citation checking naturally, because the quote is the citation.

Sample and compare

For high-stakes answers, generate three at a non-zero temperature and compare. Agreement is weak evidence of support; disagreement is strong evidence of uncertainty, and a disagreement is flagged rather than answered. Three times the cost, so reserve it for the questions where being wrong is expensive.

Narrow the scope

A model asked about one document invents less than one asked about the world. Product scope is a hallucination control: a support assistant that only answers from the manual, and says so, is more reliable than the same model with the same manual and permission to help with anything.

What the number looks like

On hard question sets without retrieval, a noticeable share of claims (sometimes a fifth or more) are unsupported. With good retrieval, abstention and citation checking, teams typically get into low single digits. Not zero. Measure it with the judge from the evaluation module: for each claim, supported, unsupported or contradicted by the context. Report the rate with its interval and track it per version.

Where not-zero is not acceptable

Medication doses, legal filings, financial figures, anything where one wrong number harms someone. The design there is that the model drafts and a person verifies each claim before it is used; and the interface must make verification easy, with the source shown beside the claim, or the person will approve without reading, which is worse than no review because it looks like one. If verification cannot be made real, the honest decision is not to ship that feature. Fine-tuning does not change this: it teaches style and format, not reliable facts, as the fine-tuning decision lesson explained.