{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"anatomy-of-a-hallucination","title":"Where invention concentrates, and why it never says \"I don't know\"","url":"https://addaly.com/learn/ai-actually-explained/anatomy-of-a-hallucination","position":32,"of":92,"minutes":9,"answer":"There is no separate inventing mode: the same next-token operation produces truth where training was dense and plausible fiction where it was thin, and invention clusters in specific details inside familiar structures.","keyIdea":"There is no separate inventing mode: the same next-token operation produces truth where training was dense and plausible fiction where it was thin, and invention clusters in specific details inside familiar structures."},"body":{"format":"markdown","text":"## There is no second mode\n\nThe common mental picture is that a model retrieves when it knows and invents when it does not. There is no such switch. It performs one operation (produce the likeliest continuation) and the quality of the output depends entirely on how much support the training gave it.\n\nWhere support is dense, the likeliest continuation is the true one. Where support is thin, the likeliest continuation is the *plausible* one. Same operation, same fluency, same confident tone. The model has no way to notice which side of that line it is on, because noticing would require a second system watching the first, and there isn't one.\n\nThis is why hallucination is not a bug that a future release removes. It is the behaviour of the mechanism at the edge of its knowledge.\n\n## Where invention clusters\n\nThe useful thing is that it clusters predictably. Six places, in rough order of danger:\n\n1. **Citations and references.** Authors, years, journals, page numbers, DOIs. The shape is learnable without any real document behind it.\n2. **URLs.** Confidently constructed and frequently dead, because a plausible web address is trivially generated.\n3. **Numbers and dates.** Statistics, populations, prices, effective dates. A specific figure looks authoritative and costs the model nothing.\n4. **Quotations.** Attributed to real people, phrased as they might have phrased it, never said.\n5. **Named minor entities.** A small company, a local regulation, a village clinic, a niche product. Enough context exists to write about it; not enough to write about it accurately.\n6. **Anything at the boundary of a real subject.** The general framework is right and the specifics inside it are invented: the most dangerous shape, because the correct scaffolding lends credibility to the fabricated detail.\n\nNotice what these have in common. Every one is a **specific detail inside a familiar structure**. That is exactly where a next-token machine is strongest at form and weakest at fact.\n\n## The specificity signal\n\nHere is a counter-intuitive tell worth internalising.\n\nInvented material is often *tidier* than real material. A real answer has ragged edges: the study had 1,247 participants, ran in two of the four planned sites, and reported a confidence interval that overlaps zero. An invented one has 1,200 participants, four sites, and a clean result.\n\nWhen a model's answer is unusually neat (round numbers, symmetric structure, every fact conveniently supporting the point) treat the neatness as a warning rather than as competence. Reality is lumpier than the likeliest continuation.\n\n## Why it guesses instead of abstaining\n\nTwo reasons, and the second is the interesting one.\n\nFirst, the training text. Documents that say \"I do not know\" are rare; documents that answer are the norm. The pattern being learned is: question, then answer.\n\nSecond, the way models are scored. A paper from OpenAI in 2025 made this argument directly: hallucination persists partly because the benchmarks used to evaluate models mark answers right or wrong with nothing in between. Under that scoring, a guess has positive expected value and an abstention has none. A model that says \"I am not sure\" scores zero; a model that guesses scores sometimes. Optimise against those scoreboards for years and you get a system that always answers.\n\nThat framing is useful because it points at a fix that is not about model architecture at all. If evaluations gave partial credit for well-placed uncertainty, the incentive would change. Some are starting to.\n\n## Three checks that cost almost nothing\n\n**Ask it three times, in three fresh conversations.** Invented specifics vary between runs, because they are sampled rather than recalled. Well-supported facts come back the same. If you get three different publication years for the same paper, none of them is real. This is the single most efficient test in this course and it takes two minutes.\n\n**Check the most specific claim first.** Not the general one. If the citation, the figure or the URL is wrong, the surrounding paragraph does not deserve trust either.\n\n**Ask for the reasoning before the answer,** and ask what would make the answer wrong. It does not make the model honest, but the second question often surfaces the conditions under which the first answer breaks down, and those conditions are frequently exactly your situation.\n\n## What not to do\n\nDo not ask \"are you sure?\" as a verification method. It is not a check on anything internal; it is a request for text about certainty, and under preference training the likely response is either warm reassurance or a reversal, depending on your tone. You will learn about the model's manners and nothing about the fact."},"check":{"question":"Which answer should be treated as most suspect, assuming the model has no search tool?","options":[{"text":"A general explanation of how monsoon rainfall affects crop planting.","correct":false,"tempting":"Picks well-documented general knowledge, which is where support is densest and invention least likely."},{"text":"A refusal to answer a question about a small local co-operative.","correct":false,"tempting":"Treats abstention as a failure, when it is the safest possible output at the edge of knowledge."},{"text":"A statement that the model is uncertain about a recent policy change.","correct":false,"tempting":"Reads hedging as unreliability; expressed uncertainty near a cut-off is appropriate rather than suspect."},{"text":"A correct-sounding overview of a district irrigation scheme, containing a named 2019 government order with a number and a date.","correct":true}]},"prev":{"slug":"confidently-wrong","title":"Why it can be wrong and sound completely sure","url":"https://addaly.com/learn/ai-actually-explained/confidently-wrong"},"next":{"slug":"what-it-cannot-do","title":"What it cannot do, honestly","url":"https://addaly.com/learn/ai-actually-explained/what-it-cannot-do"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where invention concentrates, and why it never says \"I don't know\"', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/anatomy-of-a-hallucination (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where invention concentrates, and why it never says “I don’t know”","courseTitle":"AI, Actually Explained","module":{"index":4,"slug":"why-it-goes-wrong","title":"Why it gets things wrong"},"answer":null,"keyIdea":"There is no separate inventing mode: the same next-token operation produces truth where training was dense and plausible fiction where it was thin, and invention clusters in specific details inside familiar structures.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"there-is-no-second-mode","runs":[{"text":"There is no second mode"}]},{"t":"p","runs":[{"text":"The common mental picture is that a model retrieves when it knows and invents when it does not. There is no such switch. It performs one operation (produce the likeliest continuation) and the quality of the output depends entirely on how much support the training gave it."}]},{"t":"p","runs":[{"text":"Where support is dense, the likeliest continuation is the true one. Where support is thin, the likeliest continuation is the "},{"text":"plausible","i":true},{"text":" one. Same operation, same fluency, same confident tone. The model has no way to notice which side of that line it is on, because noticing would require a second system watching the first, and there isn’t one."}]},{"t":"p","runs":[{"text":"This is why hallucination is not a bug that a future release removes. It is the behaviour of the mechanism at the edge of its knowledge."}]},{"t":"h2","id":"where-invention-clusters","runs":[{"text":"Where invention clusters"}]},{"t":"p","runs":[{"text":"The useful thing is that it clusters predictably. Six places, in rough order of danger:"}]},{"t":"ol","start":1,"items":[[{"text":"Citations and references.","b":true},{"text":" Authors, years, journals, page numbers, DOIs. The shape is learnable without any real document behind it."}],[{"text":"URLs.","b":true},{"text":" Confidently constructed and frequently dead, because a plausible web address is trivially generated."}],[{"text":"Numbers and dates.","b":true},{"text":" Statistics, populations, prices, effective dates. A specific figure looks authoritative and costs the model nothing."}],[{"text":"Quotations.","b":true},{"text":" Attributed to real people, phrased as they might have phrased it, never said."}],[{"text":"Named minor entities.","b":true},{"text":" A small company, a local regulation, a village clinic, a niche product. Enough context exists to write about it; not enough to write about it accurately."}],[{"text":"Anything at the boundary of a real subject.","b":true},{"text":" The general framework is right and the specifics inside it are invented: the most dangerous shape, because the correct scaffolding lends credibility to the fabricated detail."}]]},{"t":"p","runs":[{"text":"Notice what these have in common. Every one is a "},{"text":"specific detail inside a familiar structure","b":true},{"text":". That is exactly where a next-token machine is strongest at form and weakest at fact."}]},{"t":"h2","id":"the-specificity-signal","runs":[{"text":"The specificity signal"}]},{"t":"p","runs":[{"text":"Here is a counter-intuitive tell worth internalising."}]},{"t":"p","runs":[{"text":"Invented material is often "},{"text":"tidier","i":true},{"text":" than real material. A real answer has ragged edges: the study had 1,247 participants, ran in two of the four planned sites, and reported a confidence interval that overlaps zero. An invented one has 1,200 participants, four sites, and a clean result."}]},{"t":"p","runs":[{"text":"When a model’s answer is unusually neat (round numbers, symmetric structure, every fact conveniently supporting the point) treat the neatness as a warning rather than as competence. Reality is lumpier than the likeliest continuation."}]},{"t":"h2","id":"why-it-guesses-instead-of-abstaining","runs":[{"text":"Why it guesses instead of abstaining"}]},{"t":"p","runs":[{"text":"Two reasons, and the second is the interesting one."}]},{"t":"p","runs":[{"text":"First, the training text. Documents that say “I do not know” are rare; documents that answer are the norm. The pattern being learned is: question, then answer."}]},{"t":"p","runs":[{"text":"Second, the way models are scored. A paper from OpenAI in 2025 made this argument directly: hallucination persists partly because the benchmarks used to evaluate models mark answers right or wrong with nothing in between. Under that scoring, a guess has positive expected value and an abstention has none. A model that says “I am not sure” scores zero; a model that guesses scores sometimes. Optimise against those scoreboards for years and you get a system that always answers."}]},{"t":"p","runs":[{"text":"That framing is useful because it points at a fix that is not about model architecture at all. If evaluations gave partial credit for well-placed uncertainty, the incentive would change. Some are starting to."}]},{"t":"h2","id":"three-checks-that-cost-almost-nothing","runs":[{"text":"Three checks that cost almost nothing"}]},{"t":"p","runs":[{"text":"Ask it three times, in three fresh conversations.","b":true},{"text":" Invented specifics vary between runs, because they are sampled rather than recalled. Well-supported facts come back the same. If you get three different publication years for the same paper, none of them is real. This is the single most efficient test in this course and it takes two minutes."}]},{"t":"p","runs":[{"text":"Check the most specific claim first.","b":true},{"text":" Not the general one. If the citation, the figure or the URL is wrong, the surrounding paragraph does not deserve trust either."}]},{"t":"p","runs":[{"text":"Ask for the reasoning before the answer,","b":true},{"text":" and ask what would make the answer wrong. It does not make the model honest, but the second question often surfaces the conditions under which the first answer breaks down, and those conditions are frequently exactly your situation."}]},{"t":"h2","id":"what-not-to-do","runs":[{"text":"What not to do"}]},{"t":"p","runs":[{"text":"Do not ask “are you sure?” as a verification method. It is not a check on anything internal; it is a request for text about certainty, and under preference training the likely response is either warm reassurance or a reversal, depending on your tone. You will learn about the model’s manners and nothing about the fact."}]}],"check":{"question":[{"text":"Which answer should be treated as most suspect, assuming the model has no search tool?"}],"options":[{"text":[{"text":"A general explanation of how monsoon rainfall affects crop planting."}],"correct":false,"tempting":[{"text":"Picks well-documented general knowledge, which is where support is densest and invention least likely."}]},{"text":[{"text":"A refusal to answer a question about a small local co-operative."}],"correct":false,"tempting":[{"text":"Treats abstention as a failure, when it is the safest possible output at the edge of knowledge."}]},{"text":[{"text":"A statement that the model is uncertain about a recent policy change."}],"correct":false,"tempting":[{"text":"Reads hedging as unreliability; expressed uncertainty near a cut-off is appropriate rather than suspect."}]},{"text":[{"text":"A correct-sounding overview of a district irrigation scheme, containing a named 2019 government order with a number and a date."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"confidently-wrong","title":"Why it can be wrong and sound completely sure","minutes":7},"next":{"slug":"what-it-cannot-do","title":"What it cannot do, honestly","minutes":8}}}