{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"reducing-hallucination-by-mechanism","title":"Reducing hallucination by mechanism, not by instruction","url":"https://addaly.com/learn/building-with-ai/reducing-hallucination-by-mechanism","position":68,"of":80,"minutes":9,"answer":"Telling a model not to hallucinate does nothing measurable because it cannot tell which of its outputs are unsupported, so the rate is lowered by mechanisms (better retrieval, abstention below a score.","keyIdea":"Telling a model not to hallucinate does nothing measurable because it cannot tell which of its outputs are unsupported, so the rate is lowered by mechanisms (better retrieval, abstention below a score, claims that must cite and are checked against their source, constrained output) and it goes down, not to zero, which decides where a person must sign."},"body":{"format":"markdown","text":"## What \"do not hallucinate\" does\n\nAlmost nothing you can measure. The instruction asks the model to distinguish its supported outputs from its unsupported ones, and it has no access to that distinction. Every token is produced the same way, by the same process, whether or not the fact it completes is true. The model that writes a correct date and the model that writes a plausible wrong one are the same model doing the same thing. Asking it to stop is asking for a self-knowledge that is not there.\n\nTemperature zero does not help either. It picks the most probable token, and the most probable token can be the confident wrong one. The sampling lesson in the first module explained why.\n\nWhat does help is a set of mechanisms that change what the model is given, what it is allowed to produce, and what is checked afterwards. Here they are, roughly in order of effect.\n\n## Retrieval quality\n\nThe largest lever by a distance. A model that has the right passage in front of it invents far less than one that does not, because completing text from a source is a different task from completing it from memory. Every improvement in the retrieval module (recall, ranking, ingestion) lowers the hallucination rate before any other technique is applied. If the two-by-two from the evaluation module shows retrieval misses, start there.\n\n## Abstention below a score\n\nThe reranking lesson set a cutoff: below a certain reranker score, nothing relevant was found. Use it. When the best candidate scores below the cutoff, the answer is \"I could not find this in the available documents\", not five weak chunks and a guess. This trades some answerable questions for far fewer invented answers, and the trade is measured on the evaluation set: count the questions refused that had answers, and the questions answered wrongly that would have been refused, at each cutoff. The abstention path is a feature, and the interface should make it read as one.\n\n## Claims that cite, and citations that are checked\n\nRequire every factual sentence to carry a chunk id. Then check each one. The cheap check is existence: the id was in the retrieved set. The stronger check is support: a second, small call, or a free entailment model, asked whether chunk X supports sentence Y. Sentences that fail are removed or marked. This is claim-level verification, and it is the mechanism that turns \"grounded\" from a hope into a property. It costs one small call per answer and catches the confident sentence that cites a real chunk which says something else.\n\n## Constrained output\n\nWhere the answer space is known, make it an enumeration. A plan name from a list of four cannot be a fifth plan. A date must parse. A price must match a price in the retrieved data. The structured-output lessons built this; here its purpose is to make certain inventions impossible rather than merely unlikely.\n\n## Quote first\n\nThe grounding lesson in the prompting module asked the model to quote the relevant passage before answering. The mechanism is attention: producing the quote anchors the answer to text that exists. It is cheap and it measurably reduces unsupported claims, and it combines with citation checking naturally, because the quote is the citation.\n\n## Sample and compare\n\nFor high-stakes answers, generate three at a non-zero temperature and compare. Agreement is weak evidence of support; disagreement is strong evidence of uncertainty, and a disagreement is flagged rather than answered. Three times the cost, so reserve it for the questions where being wrong is expensive.\n\n## Narrow the scope\n\nA model asked about one document invents less than one asked about the world. Product scope is a hallucination control: a support assistant that only answers from the manual, and says so, is more reliable than the same model with the same manual and permission to help with anything.\n\n## What the number looks like\n\nOn hard question sets without retrieval, a noticeable share of claims (sometimes a fifth or more) are unsupported. With good retrieval, abstention and citation checking, teams typically get into low single digits. Not zero. Measure it with the judge from the evaluation module: for each claim, supported, unsupported or contradicted by the context. Report the rate with its interval and track it per version.\n\n## Where not-zero is not acceptable\n\nMedication doses, legal filings, financial figures, anything where one wrong number harms someone. The design there is that the model drafts and a person verifies each claim before it is used; and the interface must make verification easy, with the source shown beside the claim, or the person will approve without reading, which is worse than no review because it looks like one. If verification cannot be made real, the honest decision is not to ship that feature. Fine-tuning does not change this: it teaches style and format, not reliable facts, as the fine-tuning decision lesson explained."},"check":{"question":"Why does adding \"never state anything you are not certain of\" to a system prompt produce no measurable drop in unsupported claims?","options":[{"text":"The model cannot tell which of its outputs are unsupported; every token comes from the same process whether or not the fact is true.","correct":true},{"text":"System prompts are weaker than user messages, so the instruction is outweighed by the question.","correct":false,"tempting":"System prompts carry substantial weight; the instruction is followed as far as it can be, which is not far."},{"text":"The instruction is too vague; a precise definition of \"certain\" would allow the model to comply.","correct":false,"tempting":"Precision in wording does not create the internal signal the instruction depends on."},{"text":"It works only at temperature zero, where the model's most probable outputs are also its most reliable ones, so the sampling setting must be changed first.","correct":false,"tempting":"The most probable token is frequently the confident wrong one; temperature does not sort truth from fluency."}]},"prev":{"slug":"denial-of-wallet","title":"Denial of wallet: attacks on your bill","url":"https://addaly.com/learn/building-with-ai/denial-of-wallet"},"next":{"slug":"red-teaming-your-own-feature","title":"Red-teaming your own feature before someone else does","url":"https://addaly.com/learn/building-with-ai/red-teaming-your-own-feature"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reducing hallucination by mechanism, not by instruction', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/reducing-hallucination-by-mechanism (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reducing hallucination by mechanism, not by instruction","courseTitle":"Building with AI","module":{"index":7,"slug":"safety-security-and-the-adversary","title":"Safety, security and the adversary"},"answer":null,"keyIdea":"Telling a model not to hallucinate does nothing measurable because it cannot tell which of its outputs are unsupported, so the rate is lowered by mechanisms (better retrieval, abstention below a score, claims that must cite and are checked against their source, constrained output) and it goes down, not to zero, which decides where a person must sign.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-do-not-hallucinate-does","runs":[{"text":"What “do not hallucinate” does"}]},{"t":"p","runs":[{"text":"Almost nothing you can measure. The instruction asks the model to distinguish its supported outputs from its unsupported ones, and it has no access to that distinction. Every token is produced the same way, by the same process, whether or not the fact it completes is true. The model that writes a correct date and the model that writes a plausible wrong one are the same model doing the same thing. Asking it to stop is asking for a self-knowledge that is not there."}]},{"t":"p","runs":[{"text":"Temperature zero does not help either. It picks the most probable token, and the most probable token can be the confident wrong one. The sampling lesson in the first module explained why."}]},{"t":"p","runs":[{"text":"What does help is a set of mechanisms that change what the model is given, what it is allowed to produce, and what is checked afterwards. Here they are, roughly in order of effect."}]},{"t":"h2","id":"retrieval-quality","runs":[{"text":"Retrieval quality"}]},{"t":"p","runs":[{"text":"The largest lever by a distance. A model that has the right passage in front of it invents far less than one that does not, because completing text from a source is a different task from completing it from memory. Every improvement in the retrieval module (recall, ranking, ingestion) lowers the hallucination rate before any other technique is applied. If the two-by-two from the evaluation module shows retrieval misses, start there."}]},{"t":"h2","id":"abstention-below-a-score","runs":[{"text":"Abstention below a score"}]},{"t":"p","runs":[{"text":"The reranking lesson set a cutoff: below a certain reranker score, nothing relevant was found. Use it. When the best candidate scores below the cutoff, the answer is “I could not find this in the available documents”, not five weak chunks and a guess. This trades some answerable questions for far fewer invented answers, and the trade is measured on the evaluation set: count the questions refused that had answers, and the questions answered wrongly that would have been refused, at each cutoff. The abstention path is a feature, and the interface should make it read as one."}]},{"t":"h2","id":"claims-that-cite-and-citations-that-are-checked","runs":[{"text":"Claims that cite, and citations that are checked"}]},{"t":"p","runs":[{"text":"Require every factual sentence to carry a chunk id. Then check each one. The cheap check is existence: the id was in the retrieved set. The stronger check is support: a second, small call, or a free entailment model, asked whether chunk X supports sentence Y. Sentences that fail are removed or marked. This is claim-level verification, and it is the mechanism that turns “grounded” from a hope into a property. It costs one small call per answer and catches the confident sentence that cites a real chunk which says something else."}]},{"t":"h2","id":"constrained-output","runs":[{"text":"Constrained output"}]},{"t":"p","runs":[{"text":"Where the answer space is known, make it an enumeration. A plan name from a list of four cannot be a fifth plan. A date must parse. A price must match a price in the retrieved data. The structured-output lessons built this; here its purpose is to make certain inventions impossible rather than merely unlikely."}]},{"t":"h2","id":"quote-first","runs":[{"text":"Quote first"}]},{"t":"p","runs":[{"text":"The grounding lesson in the prompting module asked the model to quote the relevant passage before answering. The mechanism is attention: producing the quote anchors the answer to text that exists. It is cheap and it measurably reduces unsupported claims, and it combines with citation checking naturally, because the quote is the citation."}]},{"t":"h2","id":"sample-and-compare","runs":[{"text":"Sample and compare"}]},{"t":"p","runs":[{"text":"For high-stakes answers, generate three at a non-zero temperature and compare. Agreement is weak evidence of support; disagreement is strong evidence of uncertainty, and a disagreement is flagged rather than answered. Three times the cost, so reserve it for the questions where being wrong is expensive."}]},{"t":"h2","id":"narrow-the-scope","runs":[{"text":"Narrow the scope"}]},{"t":"p","runs":[{"text":"A model asked about one document invents less than one asked about the world. Product scope is a hallucination control: a support assistant that only answers from the manual, and says so, is more reliable than the same model with the same manual and permission to help with anything."}]},{"t":"h2","id":"what-the-number-looks-like","runs":[{"text":"What the number looks like"}]},{"t":"p","runs":[{"text":"On hard question sets without retrieval, a noticeable share of claims (sometimes a fifth or more) are unsupported. With good retrieval, abstention and citation checking, teams typically get into low single digits. Not zero. Measure it with the judge from the evaluation module: for each claim, supported, unsupported or contradicted by the context. Report the rate with its interval and track it per version."}]},{"t":"h2","id":"where-not-zero-is-not-acceptable","runs":[{"text":"Where not-zero is not acceptable"}]},{"t":"p","runs":[{"text":"Medication doses, legal filings, financial figures, anything where one wrong number harms someone. The design there is that the model drafts and a person verifies each claim before it is used; and the interface must make verification easy, with the source shown beside the claim, or the person will approve without reading, which is worse than no review because it looks like one. If verification cannot be made real, the honest decision is not to ship that feature. Fine-tuning does not change this: it teaches style and format, not reliable facts, as the fine-tuning decision lesson explained."}]}],"check":{"question":[{"text":"Why does adding “never state anything you are not certain of” to a system prompt produce no measurable drop in unsupported claims?"}],"options":[{"text":[{"text":"The model cannot tell which of its outputs are unsupported; every token comes from the same process whether or not the fact is true."}],"correct":true,"tempting":null},{"text":[{"text":"System prompts are weaker than user messages, so the instruction is outweighed by the question."}],"correct":false,"tempting":[{"text":"System prompts carry substantial weight; the instruction is followed as far as it can be, which is not far."}]},{"text":[{"text":"The instruction is too vague; a precise definition of “certain” would allow the model to comply."}],"correct":false,"tempting":[{"text":"Precision in wording does not create the internal signal the instruction depends on."}]},{"text":[{"text":"It works only at temperature zero, where the model’s most probable outputs are also its most reliable ones, so the sampling setting must be changed first."}],"correct":false,"tempting":[{"text":"The most probable token is frequently the confident wrong one; temperature does not sort truth from fluency."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"denial-of-wallet","title":"Denial of wallet: attacks on your bill","minutes":8},"next":{"slug":"red-teaming-your-own-feature","title":"Red-teaming your own feature before someone else does","minutes":9}}}