{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"hallucination-and-verification","title":"Hallucination, and how to check","url":"https://addaly.com/learn/ai-and-you/hallucination-and-verification","position":18,"of":73,"minutes":8,"answer":"A model produces wrong answers in exactly the same confident voice as right ones; check consequences, not tone.","keyIdea":"A model produces wrong answers in exactly the same confident voice as right ones; check consequences, not tone."},"body":{"format":"markdown","text":"A language model predicts likely next words. That is the whole mechanism. It is astonishingly powerful and it explains the central failure: the model will produce a fluent, well-formed, completely fabricated answer, and it will sound exactly like the true ones.\n\nThe word \"hallucination\" is a bit misleading. The model is doing the same thing when it is right and when it is wrong.\n\n## What it looks like\n\nIn 2023, two New York lawyers submitted a court filing that cited six cases that did not exist. The model had produced case names, docket numbers, quotes and internal citations, all in the correct format. The judge fined them $5,000.\n\nFabrication clusters in predictable places:\n\n- **Citations, references and URLs.** The format is highly patterned, so it is easy to generate and hard to eyeball as fake.\n- **Numbers and dates.** Population figures, drug dosages, exchange rates, historical years.\n- **Names attached to facts.** Who founded what, who said which quote, who wrote which paper.\n- **Anything niche.** The less a topic appeared in training data, the more the model is improvising. A question about a small town's municipal rules is far riskier than one about photosynthesis.\n- **Anything recent.** Models have a training cutoff. Ask about last month and you may get a confident answer built from the shape of older events.\n\n## The confidence trap\n\nAsking \"are you sure?\" does very little. The model will often apologise and change its answer whether or not the original was correct: it is predicting what a helpful response to doubt looks like, not re-examining evidence. Some systems now show confidence scores or hedging language, and these are weakly informative at best.\n\nTreat stated confidence as a writing style, not as evidence.\n\n## How to actually check\n\nA working method, in order of effort:\n\n**Ask for the source, then open it.** Not \"give me a source\": that invites a plausible-looking fabricated one. Copy the exact title into a search engine and see whether it exists and whether it says what was claimed. A real paper that says something different is as much a failure as a fake one.\n\n**Ask the same question twice in separate conversations.** Facts the model actually holds tend to come back stable. Fabrications drift: different year, different name. This is a cheap smoke test, not proof.\n\n**Give it the document instead of asking from memory.** Paste the annual report and ask questions about the text in front of it. Grounding the model in supplied material sharply reduces invention, though it does not eliminate it: models still misread and over-summarise. (Consider the privacy cost first; see [Following the paragraph](/learn/ai-and-you/where-the-copy-goes).)\n\n**Check the claim, not the paragraph.** Fluent text carries wrong facts inside right ones. Pull out each checkable assertion separately.\n\n## The rule worth keeping\n\nMatch your verification to the consequence of being wrong. Brainstorming names for a shop: no verification needed. Drafting an email: skim it. A dosage, a legal deadline, a figure going into a report your manager will sign, a claim you will state publicly: verify every specific, from a source you opened yourself."},"check":{"question":"You ask a model for statistics on youth unemployment in Kenya. It gives a precise figure, names a report, and adds \"I am confident in this figure.\" You ask \"are you certain?\" and it confirms. What have you learned?","options":[{"text":"The figure is likely reliable, since it held up when you challenged it directly","correct":false,"tempting":"Consistency under challenge is a signal in people. The model is predicting how a confident reply continues, not re-examining a source."},{"text":"Almost nothing: it is writing text that fits a confident answer, not checking","correct":true},{"text":"The figure is probably wrong, because models are unreliable with statistics of any kind","correct":false,"tempting":"Blanket distrust feels safe, but it is just as unfounded, and wastes a tool that is often right. The reply carries no information either way."},{"text":"The named report must be real, since inventing a whole report title is unlikely","correct":false,"tempting":"A specific title feels as if it must come from somewhere. Citation formats are patterned and among the easiest things to generate."}],"why":"Confidence and confirmation come from the same process as the figure. Only the named report, opened by you, can check it."},"prev":{"slug":"the-feedback-loop-in-data","title":"The outcomes you never get to see","url":"https://addaly.com/learn/ai-and-you/the-feedback-loop-in-data"},"next":{"slug":"the-model-cannot-cite-itself","title":"Why it cannot tell you where it got that","url":"https://addaly.com/learn/ai-and-you/the-model-cannot-cite-itself"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Hallucination, and how to check', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/hallucination-and-verification (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Hallucination, and how to check","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":3,"slug":"truth-and-verification","title":"Truth, sources and verification"},"answer":null,"keyIdea":"A model produces wrong answers in exactly the same confident voice as right ones; check consequences, not tone.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"p","runs":[{"text":"A language model predicts likely next words. That is the whole mechanism. It is astonishingly powerful and it explains the central failure: the model will produce a fluent, well-formed, completely fabricated answer, and it will sound exactly like the true ones."}],"lede":true},{"t":"p","runs":[{"text":"The word “hallucination” is a bit misleading. The model is doing the same thing when it is right and when it is wrong."}]},{"t":"h2","id":"what-it-looks-like","runs":[{"text":"What it looks like"}]},{"t":"p","runs":[{"text":"In 2023, two New York lawyers submitted a court filing that cited six cases that did not exist. The model had produced case names, docket numbers, quotes and internal citations, all in the correct format. The judge fined them $5,000."}]},{"t":"p","runs":[{"text":"Fabrication clusters in predictable places:"}]},{"t":"ul","items":[[{"text":"Citations, references and URLs.","b":true},{"text":" The format is highly patterned, so it is easy to generate and hard to eyeball as fake."}],[{"text":"Numbers and dates.","b":true},{"text":" Population figures, drug dosages, exchange rates, historical years."}],[{"text":"Names attached to facts.","b":true},{"text":" Who founded what, who said which quote, who wrote which paper."}],[{"text":"Anything niche.","b":true},{"text":" The less a topic appeared in training data, the more the model is improvising. A question about a small town’s municipal rules is far riskier than one about photosynthesis."}],[{"text":"Anything recent.","b":true},{"text":" Models have a training cutoff. Ask about last month and you may get a confident answer built from the shape of older events."}]]},{"t":"h2","id":"the-confidence-trap","runs":[{"text":"The confidence trap"}]},{"t":"p","runs":[{"text":"Asking “are you sure?” does very little. The model will often apologise and change its answer whether or not the original was correct: it is predicting what a helpful response to doubt looks like, not re-examining evidence. Some systems now show confidence scores or hedging language, and these are weakly informative at best."}]},{"t":"p","runs":[{"text":"Treat stated confidence as a writing style, not as evidence."}]},{"t":"h2","id":"how-to-actually-check","runs":[{"text":"How to actually check"}]},{"t":"p","runs":[{"text":"A working method, in order of effort:"}]},{"t":"p","runs":[{"text":"Ask for the source, then open it.","b":true},{"text":" Not “give me a source”: that invites a plausible-looking fabricated one. Copy the exact title into a search engine and see whether it exists and whether it says what was claimed. A real paper that says something different is as much a failure as a fake one."}]},{"t":"p","runs":[{"text":"Ask the same question twice in separate conversations.","b":true},{"text":" Facts the model actually holds tend to come back stable. Fabrications drift: different year, different name. This is a cheap smoke test, not proof."}]},{"t":"p","runs":[{"text":"Give it the document instead of asking from memory.","b":true},{"text":" Paste the annual report and ask questions about the text in front of it. Grounding the model in supplied material sharply reduces invention, though it does not eliminate it: models still misread and over-summarise. (Consider the privacy cost first; see "},{"text":"Following the paragraph","href":"https://addaly.com/learn/ai-and-you/where-the-copy-goes"},{"text":".)"}]},{"t":"p","runs":[{"text":"Check the claim, not the paragraph.","b":true},{"text":" Fluent text carries wrong facts inside right ones. Pull out each checkable assertion separately."}]},{"t":"h2","id":"the-rule-worth-keeping","runs":[{"text":"The rule worth keeping"}]},{"t":"p","runs":[{"text":"Match your verification to the consequence of being wrong. Brainstorming names for a shop: no verification needed. Drafting an email: skim it. A dosage, a legal deadline, a figure going into a report your manager will sign, a claim you will state publicly: verify every specific, from a source you opened yourself."}]}],"check":{"question":[{"text":"You ask a model for statistics on youth unemployment in Kenya. It gives a precise figure, names a report, and adds “I am confident in this figure.” You ask “are you certain?” and it confirms. What have you learned?"}],"options":[{"text":[{"text":"The figure is likely reliable, since it held up when you challenged it directly"}],"correct":false,"tempting":[{"text":"Consistency under challenge is a signal in people. The model is predicting how a confident reply continues, not re-examining a source."}]},{"text":[{"text":"Almost nothing: it is writing text that fits a confident answer, not checking"}],"correct":true,"tempting":null},{"text":[{"text":"The figure is probably wrong, because models are unreliable with statistics of any kind"}],"correct":false,"tempting":[{"text":"Blanket distrust feels safe, but it is just as unfounded, and wastes a tool that is often right. The reply carries no information either way."}]},{"text":[{"text":"The named report must be real, since inventing a whole report title is unlikely"}],"correct":false,"tempting":[{"text":"A specific title feels as if it must come from somewhere. Citation formats are patterned and among the easiest things to generate."}]}],"order":[0,2,3,1],"why":[{"text":"Confidence and confirmation come from the same process as the figure. Only the named report, opened by you, can check it."}]},"prev":{"slug":"the-feedback-loop-in-data","title":"The outcomes you never get to see","minutes":8},"next":{"slug":"the-model-cannot-cite-itself","title":"Why it cannot tell you where it got that","minutes":9}}}