{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"hallucination-by-design","title":"Why hallucination is a property of the method","url":"https://addaly.com/learn/how-llms-work/hallucination-by-design","position":49,"of":78,"minutes":8,"answer":"The model always outputs a plausible continuation; nothing in the mechanism checks whether it is true.","keyIdea":"The model always outputs a plausible continuation; nothing in the mechanism checks whether it is true."},"body":{"format":"markdown","text":"## There is no truth step\n\nWalk back through the whole pipeline. Text becomes tokens. Tokens become vectors. Blocks add contributions. The final vector produces a score for every possible next token. One is sampled.\n\nNowhere in that sequence is there a check against the world. There is no database lookup, no verification, no flag that fires when the model reaches beyond what it saw. The machinery that produces a correct fact and the machinery that produces a fabricated one are the same machinery, running identically. From the inside, nothing distinguishes them.\n\n> Figure: The whole pipeline, with the missing step named. Steps: 1. Text becomes tokens (A compression rule frozen on one corpus); 2. Tokens become vectors (A lookup row per token, rewritten as context arrives); 3. Blocks add contributions (Attention gathers from earlier positions, feed-forward looks things up); 4. A score for every possible next token (One comparison per vocabulary entry); 5. One token is sampled (The same machinery whether the fact is real or invented); 6. No step checks the world (No database lookup, no verification, no flag that fires when the model reaches past what it saw). This is why the word hallucination misleads. Nothing is malfunctioning: the system produces the plausible continuation on an input where the plausible continuation happens to be false. A fabricated citation is as well-formed as a real one because well-formedness is what the objective optimised.\n\nThis is why \"hallucination\" is a slightly misleading word. It suggests a malfunction. What actually happens is the system doing precisely what it does, on an input where the plausible continuation happens not to be true.\n\n## Why the invented details look so good\n\nA fabricated citation arrives with a plausible journal, a volume number, a year, a page range and authors who work in that field. That is not the model trying to deceive you. It is the model producing the shape of a citation, because shape is exactly what it learned. A real citation and a fake one are equally well-formed, so well-formedness cannot separate them, and well-formedness is what the objective optimised.\n\nThe same explains why hallucination clusters where it does. Common facts appear thousands of times in training and are strongly represented. Facts that appear twice (the exact filing date of one court case, a specific figure in one company's report, the API signature of a small library) leave a weak trace, and something plausible fills the gap.\n\n## Saying \"I don't know\" is just another token sequence\n\nThere is no abstain button. \"I am not sure\" is a string of tokens that must out-score the fluent answer. Whether it does is decided by the same distribution as everything else.\n\nAnd training pushes against it. One influential argument holds that the way models are evaluated is much of the cause: benchmarks score an answer as right or wrong, a blank gets zero, and a guess sometimes gets a point. Under that scoring, a model that guesses beats an identical model that abstains, so guessing is what gets selected for, stage after stage. Preference training pulls the same way, since raters tend to prefer confident answers to hedged ones. This account is argued rather than proven, but the incentive it describes is straightforwardly real.\n\nThere is a related and better-established finding: models often carry an internal signal that correlates with whether they are about to be wrong. The information exists somewhere in the network. It is just not what governs the output.\n\n## What actually helps, honestly rated\n\n**Retrieval.** Put the source text in the prompt and ask for an answer from it. This is the largest single improvement available, and it is not a cure: models still drift beyond the provided passage, and retrieving the wrong passage produces a confident answer about the wrong thing.\n\n**Verified citations.** Asking for citations without giving the model a search tool asks for well-shaped citations, which is what you will get. Citations are only worth trusting when something actually fetched the document.\n\n**Sampling several times.** Ask the same question five times at moderate temperature. Agreement is weak evidence of solid ground; contradiction is strong evidence you are in thin material. Crude, cheap, effective.\n\n**A separate checking pass.** Giving the answer and the source back to a model and asking whether each claim is supported catches a real fraction of errors, because judging support is an easier task than generating from nothing.\n\nWhat does not help: telling the model not to hallucinate. It has no lever for that.\n\n## The design consequence\n\nRates fall with better models and better grounding, and they do not reach zero, because nothing in the method contains a step that could make them zero. So build accordingly. In any workflow where a false confident answer is expensive (medical, legal, financial, safety) the model drafts and a human or a deterministic system verifies. That is not a temporary precaution for the current generation of models. It follows from how the thing works."},"check":{"question":"A model produces a plausible-looking academic citation with a real-sounding journal, a volume number and page numbers. The paper does not exist. Which description is most accurate?","options":[{"text":"It knew it had no real source and covered for itself, the way a person bluffs in a meeting.","correct":false,"tempting":"The output is indistinguishable from human bluffing, and intent is the natural explanation people reach for when confronted with confident falsehood."},{"text":"That citation exists somewhere in the training data and was recalled imperfectly.","correct":false,"tempting":"Treating the model as a lossy database explains many real cases and feels mechanical rather than mystical."},{"text":"It is a failure mode of smaller models; the largest models have effectively solved it.","correct":false,"tempting":"Bigger models genuinely do hallucinate less on common facts, so the trend is real and extrapolating it is reasonable."},{"text":"It produced the token sequence that best fits the shape of a citation in that context, and nothing in the process consulted any record of what exists.","correct":true}]},"prev":{"slug":"emergence-and-the-argument","title":"Emergent abilities, and the argument about whether they exist","url":"https://addaly.com/learn/how-llms-work/emergence-and-the-argument"},"next":{"slug":"measuring-uncertainty","title":"Measuring what the model does not know","url":"https://addaly.com/learn/how-llms-work/measuring-uncertainty"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why hallucination is a property of the method', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/hallucination-by-design (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why hallucination is a property of the method","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"The model always outputs a plausible continuation; nothing in the mechanism checks whether it is true.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 6.1","figure":{"type":"flow","steps":[{"label":"Text becomes tokens","detail":"A compression rule frozen on one corpus"},{"label":"Tokens become vectors","detail":"A lookup row per token, rewritten as context arrives"},{"label":"Blocks add contributions","detail":"Attention gathers from earlier positions, feed-forward looks things up"},{"label":"A score for every possible next token","detail":"One comparison per vocabulary entry"},{"label":"One token is sampled","detail":"The same machinery whether the fact is real or invented"},{"label":"No step checks the world","detail":"No database lookup, no verification, no flag that fires when the model reaches past what it saw"}],"title":"The whole pipeline, with the missing step named","caption":"This is why the word hallucination misleads. Nothing is malfunctioning: the system produces the plausible continuation on an input where the plausible continuation happens to be false. A fabricated citation is as well-formed as a real one because well-formedness is what the objective optimised."},"alt":"Figure: The whole pipeline, with the missing step named. Steps: 1. Text becomes tokens (A compression rule frozen on one corpus); 2. Tokens become vectors (A lookup row per token, rewritten as context arrives); 3. Blocks add contributions (Attention gathers from earlier positions, feed-forward looks things up); 4. A score for every possible next token (One comparison per vocabulary entry); 5. One token is sampled (The same machinery whether the fact is real or invented); 6. No step checks the world (No database lookup, no verification, no flag that fires when the model reaches past what it saw). This is why the word hallucination misleads. Nothing is malfunctioning: the system produces the plausible continuation on an input where the plausible continuation happens to be false. A fabricated citation is as well-formed as a real one because well-formedness is what the objective optimised."}},"blocks":[{"t":"h2","id":"there-is-no-truth-step","runs":[{"text":"There is no truth step"}]},{"t":"p","runs":[{"text":"Walk back through the whole pipeline. Text becomes tokens. Tokens become vectors. Blocks add contributions. The final vector produces a score for every possible next token. One is sampled."}]},{"t":"p","runs":[{"text":"Nowhere in that sequence is there a check against the world. There is no database lookup, no verification, no flag that fires when the model reaches beyond what it saw. The machinery that produces a correct fact and the machinery that produces a fabricated one are the same machinery, running identically. From the inside, nothing distinguishes them."}]},{"t":"p","runs":[{"text":"This is why “hallucination” is a slightly misleading word. It suggests a malfunction. What actually happens is the system doing precisely what it does, on an input where the plausible continuation happens not to be true."}]},{"t":"h2","id":"why-the-invented-details-look-so-good","runs":[{"text":"Why the invented details look so good"}]},{"t":"p","runs":[{"text":"A fabricated citation arrives with a plausible journal, a volume number, a year, a page range and authors who work in that field. That is not the model trying to deceive you. It is the model producing the shape of a citation, because shape is exactly what it learned. A real citation and a fake one are equally well-formed, so well-formedness cannot separate them, and well-formedness is what the objective optimised."}]},{"t":"p","runs":[{"text":"The same explains why hallucination clusters where it does. Common facts appear thousands of times in training and are strongly represented. Facts that appear twice (the exact filing date of one court case, a specific figure in one company’s report, the API signature of a small library) leave a weak trace, and something plausible fills the gap."}]},{"t":"h2","id":"saying-i-don-t-know-is-just-another-token-sequence","runs":[{"text":"Saying “I don’t know” is just another token sequence"}]},{"t":"p","runs":[{"text":"There is no abstain button. “I am not sure” is a string of tokens that must out-score the fluent answer. Whether it does is decided by the same distribution as everything else."}]},{"t":"p","runs":[{"text":"And training pushes against it. One influential argument holds that the way models are evaluated is much of the cause: benchmarks score an answer as right or wrong, a blank gets zero, and a guess sometimes gets a point. Under that scoring, a model that guesses beats an identical model that abstains, so guessing is what gets selected for, stage after stage. Preference training pulls the same way, since raters tend to prefer confident answers to hedged ones. This account is argued rather than proven, but the incentive it describes is straightforwardly real."}]},{"t":"p","runs":[{"text":"There is a related and better-established finding: models often carry an internal signal that correlates with whether they are about to be wrong. The information exists somewhere in the network. It is just not what governs the output."}]},{"t":"h2","id":"what-actually-helps-honestly-rated","runs":[{"text":"What actually helps, honestly rated"}]},{"t":"p","runs":[{"text":"Retrieval.","b":true},{"text":" Put the source text in the prompt and ask for an answer from it. This is the largest single improvement available, and it is not a cure: models still drift beyond the provided passage, and retrieving the wrong passage produces a confident answer about the wrong thing."}]},{"t":"p","runs":[{"text":"Verified citations.","b":true},{"text":" Asking for citations without giving the model a search tool asks for well-shaped citations, which is what you will get. Citations are only worth trusting when something actually fetched the document."}]},{"t":"p","runs":[{"text":"Sampling several times.","b":true},{"text":" Ask the same question five times at moderate temperature. Agreement is weak evidence of solid ground; contradiction is strong evidence you are in thin material. Crude, cheap, effective."}]},{"t":"p","runs":[{"text":"A separate checking pass.","b":true},{"text":" Giving the answer and the source back to a model and asking whether each claim is supported catches a real fraction of errors, because judging support is an easier task than generating from nothing."}]},{"t":"p","runs":[{"text":"What does not help: telling the model not to hallucinate. It has no lever for that."}]},{"t":"h2","id":"the-design-consequence","runs":[{"text":"The design consequence"}]},{"t":"p","runs":[{"text":"Rates fall with better models and better grounding, and they do not reach zero, because nothing in the method contains a step that could make them zero. So build accordingly. In any workflow where a false confident answer is expensive (medical, legal, financial, safety) the model drafts and a human or a deterministic system verifies. That is not a temporary precaution for the current generation of models. It follows from how the thing works."}]}],"check":{"question":[{"text":"A model produces a plausible-looking academic citation with a real-sounding journal, a volume number and page numbers. The paper does not exist. Which description is most accurate?"}],"options":[{"text":[{"text":"It knew it had no real source and covered for itself, the way a person bluffs in a meeting."}],"correct":false,"tempting":[{"text":"The output is indistinguishable from human bluffing, and intent is the natural explanation people reach for when confronted with confident falsehood."}]},{"text":[{"text":"That citation exists somewhere in the training data and was recalled imperfectly."}],"correct":false,"tempting":[{"text":"Treating the model as a lossy database explains many real cases and feels mechanical rather than mystical."}]},{"text":[{"text":"It is a failure mode of smaller models; the largest models have effectively solved it."}],"correct":false,"tempting":[{"text":"Bigger models genuinely do hallucinate less on common facts, so the trend is real and extrapolating it is reasonable."}]},{"text":[{"text":"It produced the token sequence that best fits the shape of a citation in that context, and nothing in the process consulted any record of what exists."}],"correct":true,"tempting":null}],"order":[2,0,1,3],"why":null},"prev":{"slug":"emergence-and-the-argument","title":"Emergent abilities, and the argument about whether they exist","minutes":8},"next":{"slug":"measuring-uncertainty","title":"Measuring what the model does not know","minutes":9}}}