{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"when-the-model-does-not-know","title":"When the model does not know","url":"https://addaly.com/learn/open-models/when-the-model-does-not-know","position":50,"of":71,"minutes":9,"answer":"A model always produces its most likely continuation, so abstention must be made available in the schema and rewarded in the prompt.","keyIdea":"A model always produces its most likely continuation, so abstention must be made available in the schema and rewarded in the prompt, and requiring a verifiable quotation turns invisible fabrication into a check that code can fail."},"body":{"format":"markdown","text":"## There is no empty answer\n\nA language model produces the most likely continuation of its context. There is no state in which it produces nothing. If the answer is not available, the most likely continuation is a plausible-looking answer, because that is what text answering a question looks like.\n\nThis is why *hallucination* is a slightly misleading word. The model is not malfunctioning; it is doing exactly what it does, on an input where the correct behaviour would have been to stop. Saying *I do not know* is a specific behaviour that has to be trained in and then made likely by the prompt and the format, and small models have less of it.\n\n## Confidence is only weakly informative\n\nThe obvious idea is to read the model's own probabilities. It half works.\n\nToken probabilities correlate with correctness, but weakly and inconsistently, and instruction tuning makes it worse: preference tuning rewards confident-sounding answers, which pushes probabilities up regardless of correctness. A model can assign 0.94 to a fabricated citation.\n\nWhere logprobs are useful is on **constrained** outputs. If the model must choose between five enum values, the probability mass over those five is meaningful, and a top choice at 0.4 against a runner-up at 0.35 is a real signal that this item needs a human. For free-form text, treat logprobs as a weak prior, not a confidence score.\n\n## Five things that work, in order of return\n\n**1. Give it the information.** Most confabulation is a knowledge problem, and retrieval solves knowledge problems that no model size solves reliably. A 4B model with the right paragraph beats a 70B model guessing, and it is cheaper.\n\n**2. Make abstention available in the format.** If the schema requires an answer, the model must invent one. Give it somewhere to go:\n\n```json\n{\"answer\": {\"type\": [\"string\",\"null\"]},\n \"evidence_quote\": {\"type\": [\"string\",\"null\"]},\n \"status\": {\"enum\": [\"answered\",\"not_in_source\",\"ambiguous\"]}}\n```\n\nThen say plainly in the prompt that `not_in_source` is a correct and expected answer. Small models need explicit permission; without it, abstention has essentially zero probability.\n\n**3. Require a quotation.** Ask for the exact span from the source that supports the answer, and verify in code that the span appears in the source. This is a hard check, not a soft one (a fabricated quote fails a string search) and it converts an invisible failure into a caught one. It is the single most effective technique in this list after retrieval.\n\n**4. Self-consistency.** Sample five completions at temperature 0.7 and compare. Agreement across samples correlates with correctness considerably better than any single answer's confidence. Disagreement is a reliable flag for review. It costs five times as much, so apply it to the items that matter rather than to everything.\n\n**5. A verification pass.** A second call, or a second model, given the source and the proposed answer and asked only whether the source supports it. Cheaper than it sounds, because the verification prompt is short and the answer is one token.\n\n## Measure it, with an unanswerable set\n\nBuild a small set that makes fabrication visible:\n\n- 20 questions whose answers **are** in your documents.\n- 20 questions that **look** answerable but are not: a plausible field that the document does not contain, a person not mentioned, a date not given.\n\nScore two numbers separately: accuracy on the answerable half, and abstention rate on the unanswerable half. A model at 90% accuracy that abstains on 10% of the unanswerable questions is worse for most products than one at 82% accuracy that abstains on 85%, because the second one's errors are visible.\n\n> Figure: Two systems on twenty answerable and twenty unanswerable questions. A two by two grid, rows System A and System B, columns Accuracy on the answerable 20 and Abstention on the unanswerable 20: System A and Accuracy on the answerable 20: 90%; System A and Abstention on the unanswerable 20: 10%; System B and Accuracy on the answerable 20: 82%; System B and Abstention on the unanswerable 20: 85%. System A is eight points better on the questions it can answer and produces eighteen confident inventions on the ones it cannot. The second column is the number nobody measures, and it is the one that decides whether people can trust the system.\n\nThis second number is the one nobody measures and the one that decides whether people can trust the system.\n\n## Say it in the interface\n\nThe final piece is not technical. If the system can be wrong, the interface should show its source and make checking easy: the quotation, a link, the document name. A confident sentence with no provenance invites trust the system has not earned; the same sentence with the supporting line beside it invites the reader to do the one check that catches the failure."},"check":{"question":"Two configurations of a document question-answering system: A answers 90% of answerable questions correctly and abstains on 10% of unanswerable ones; B answers 82% correctly and abstains on 85% of unanswerable ones. Which is usually better in production, and why?","options":[{"text":"A, because accuracy on answerable questions is the metric users experience.","correct":false,"tempting":"Users also experience the confident wrong answers on unanswerable questions, which A produces 90% of the time."},{"text":"B, because its failures mostly surface as an admission of not knowing rather than as confident fabrications the reader cannot detect.","correct":true},{"text":"A, because B's high abstention rate means it will also abstain on answerable questions and frustrate users.","correct":false,"tempting":"Assumes the abstention is untargeted, but B's answerable accuracy of 82% is measured separately and shows the cost is bounded."},{"text":"Neither; both need a confidence threshold on token probabilities to decide when to answer.","correct":false,"tempting":"Token probabilities in free-form output are a weak and poorly calibrated signal, which is why schema-level abstention is used instead."}]},"prev":{"slug":"guardrails-as-models","title":"Guardrails as a separate layer","url":"https://addaly.com/learn/open-models/guardrails-as-models"},"next":{"slug":"when-not-to-fine-tune","title":"When not to fine-tune","url":"https://addaly.com/learn/open-models/when-not-to-fine-tune"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the model does not know', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/when-the-model-does-not-know (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the model does not know","courseTitle":"The Open Model Ecosystem","module":{"index":5,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"A model always produces its most likely continuation, so abstention must be made available in the schema and rewarded in the prompt, and requiring a verifiable quotation turns invisible fabrication into a check that code can fail.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Accuracy on the answerable 20","Abstention on the unanswerable 20"],"rows":["System A","System B"],"type":"matrix","cells":[["90%","10%"],["82%","85%"]],"title":"Two systems on twenty answerable and twenty unanswerable questions","caption":"System A is eight points better on the questions it can answer and produces eighteen confident inventions on the ones it cannot. The second column is the number nobody measures, and it is the one that decides whether people can trust the system.","cellLabels":[["18 of 20 right","18 confident inventions"],["16 of 20 right","3 inventions"]]},"alt":"Figure: Two systems on twenty answerable and twenty unanswerable questions. A two by two grid, rows System A and System B, columns Accuracy on the answerable 20 and Abstention on the unanswerable 20: System A and Accuracy on the answerable 20: 90%; System A and Abstention on the unanswerable 20: 10%; System B and Accuracy on the answerable 20: 82%; System B and Abstention on the unanswerable 20: 85%. System A is eight points better on the questions it can answer and produces eighteen confident inventions on the ones it cannot. The second column is the number nobody measures, and it is the one that decides whether people can trust the system."}},"blocks":[{"t":"h2","id":"there-is-no-empty-answer","runs":[{"text":"There is no empty answer"}]},{"t":"p","runs":[{"text":"A language model produces the most likely continuation of its context. There is no state in which it produces nothing. If the answer is not available, the most likely continuation is a plausible-looking answer, because that is what text answering a question looks like."}]},{"t":"p","runs":[{"text":"This is why "},{"text":"hallucination","i":true},{"text":" is a slightly misleading word. The model is not malfunctioning; it is doing exactly what it does, on an input where the correct behaviour would have been to stop. Saying "},{"text":"I do not know","i":true},{"text":" is a specific behaviour that has to be trained in and then made likely by the prompt and the format, and small models have less of it."}]},{"t":"h2","id":"confidence-is-only-weakly-informative","runs":[{"text":"Confidence is only weakly informative"}]},{"t":"p","runs":[{"text":"The obvious idea is to read the model’s own probabilities. It half works."}]},{"t":"p","runs":[{"text":"Token probabilities correlate with correctness, but weakly and inconsistently, and instruction tuning makes it worse: preference tuning rewards confident-sounding answers, which pushes probabilities up regardless of correctness. A model can assign 0.94 to a fabricated citation."}]},{"t":"p","runs":[{"text":"Where logprobs are useful is on "},{"text":"constrained","b":true},{"text":" outputs. If the model must choose between five enum values, the probability mass over those five is meaningful, and a top choice at 0.4 against a runner-up at 0.35 is a real signal that this item needs a human. For free-form text, treat logprobs as a weak prior, not a confidence score."}]},{"t":"h2","id":"five-things-that-work-in-order-of-return","runs":[{"text":"Five things that work, in order of return"}]},{"t":"p","runs":[{"text":"1. Give it the information.","b":true},{"text":" Most confabulation is a knowledge problem, and retrieval solves knowledge problems that no model size solves reliably. A 4B model with the right paragraph beats a 70B model guessing, and it is cheaper."}]},{"t":"p","runs":[{"text":"2. Make abstention available in the format.","b":true},{"text":" If the schema requires an answer, the model must invent one. Give it somewhere to go:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"answer\": {\"type\": [\"string\",\"null\"]},\n \"evidence_quote\": {\"type\": [\"string\",\"null\"]},\n \"status\": {\"enum\": [\"answered\",\"not_in_source\",\"ambiguous\"]}}"},{"t":"p","runs":[{"text":"Then say plainly in the prompt that "},{"text":"not_in_source","code":true},{"text":" is a correct and expected answer. Small models need explicit permission; without it, abstention has essentially zero probability."}]},{"t":"p","runs":[{"text":"3. Require a quotation.","b":true},{"text":" Ask for the exact span from the source that supports the answer, and verify in code that the span appears in the source. This is a hard check, not a soft one (a fabricated quote fails a string search) and it converts an invisible failure into a caught one. It is the single most effective technique in this list after retrieval."}]},{"t":"p","runs":[{"text":"4. Self-consistency.","b":true},{"text":" Sample five completions at temperature 0.7 and compare. Agreement across samples correlates with correctness considerably better than any single answer’s confidence. Disagreement is a reliable flag for review. It costs five times as much, so apply it to the items that matter rather than to everything."}]},{"t":"p","runs":[{"text":"5. A verification pass.","b":true},{"text":" A second call, or a second model, given the source and the proposed answer and asked only whether the source supports it. Cheaper than it sounds, because the verification prompt is short and the answer is one token."}]},{"t":"h2","id":"measure-it-with-an-unanswerable-set","runs":[{"text":"Measure it, with an unanswerable set"}]},{"t":"p","runs":[{"text":"Build a small set that makes fabrication visible:"}]},{"t":"ul","items":[[{"text":"20 questions whose answers "},{"text":"are","b":true},{"text":" in your documents."}],[{"text":"20 questions that "},{"text":"look","b":true},{"text":" answerable but are not: a plausible field that the document does not contain, a person not mentioned, a date not given."}]]},{"t":"p","runs":[{"text":"Score two numbers separately: accuracy on the answerable half, and abstention rate on the unanswerable half. A model at 90% accuracy that abstains on 10% of the unanswerable questions is worse for most products than one at 82% accuracy that abstains on 85%, because the second one’s errors are visible."}]},{"t":"p","runs":[{"text":"This second number is the one nobody measures and the one that decides whether people can trust the system."}]},{"t":"h2","id":"say-it-in-the-interface","runs":[{"text":"Say it in the interface"}]},{"t":"p","runs":[{"text":"The final piece is not technical. If the system can be wrong, the interface should show its source and make checking easy: the quotation, a link, the document name. A confident sentence with no provenance invites trust the system has not earned; the same sentence with the supporting line beside it invites the reader to do the one check that catches the failure."}]}],"check":{"question":[{"text":"Two configurations of a document question-answering system: A answers 90% of answerable questions correctly and abstains on 10% of unanswerable ones; B answers 82% correctly and abstains on 85% of unanswerable ones. Which is usually better in production, and why?"}],"options":[{"text":[{"text":"A, because accuracy on answerable questions is the metric users experience."}],"correct":false,"tempting":[{"text":"Users also experience the confident wrong answers on unanswerable questions, which A produces 90% of the time."}]},{"text":[{"text":"B, because its failures mostly surface as an admission of not knowing rather than as confident fabrications the reader cannot detect."}],"correct":true,"tempting":null},{"text":[{"text":"A, because B’s high abstention rate means it will also abstain on answerable questions and frustrate users."}],"correct":false,"tempting":[{"text":"Assumes the abstention is untargeted, but B’s answerable accuracy of 82% is measured separately and shows the cost is bounded."}]},{"text":[{"text":"Neither; both need a confidence threshold on token probabilities to decide when to answer."}],"correct":false,"tempting":[{"text":"Token probabilities in free-form output are a weak and poorly calibrated signal, which is why schema-level abstention is used instead."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"guardrails-as-models","title":"Guardrails as a separate layer","minutes":8},"next":{"slug":"when-not-to-fine-tune","title":"When not to fine-tune","minutes":8}}}