{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"confidently-wrong","title":"Why it can be wrong and sound completely sure","url":"https://addaly.com/learn/ai-actually-explained/confidently-wrong","position":31,"of":92,"minutes":7,"answer":"Nothing in the model checks anything, and invention clusters exactly where the details get specific.","keyIdea":"Nothing in the model checks anything, and invention clusters exactly where the details get specific."},"body":{"format":"markdown","text":"## There is no checking step\n\nHere is the uncomfortable part of the last lesson, stated plainly.\n\nProducing a true sentence and producing a false one use exactly the same machinery. The model assembles likely-sounding continuations. Most of the time, likely-sounding and true line up, because the text it was trained on was mostly written by people trying to say true things. When they come apart, nothing notices, because there is nothing whose job is to notice.\n\nThis is usually called hallucination. That word is a bit flattering. The model is not malfunctioning when it invents a citation. It is doing precisely what it always does.\n\n## What it looks like in practice\n\nIn 2023 two lawyers in New York filed a court brief containing six cases that did not exist. A chatbot had produced them: plausible names, plausible courts, plausible years, plausible quotations. The lawyers were fined. The cases had the exact texture of real citations, because the model had absorbed the texture of thousands of real ones.\n\nThe same thing happens with a bus route that sounds right for a city it has read about, a customer helpline number with the correct number of digits, a section of a country's tax code with a convincing number, a research paper attributed to a real author who never wrote it.\n\n## Where it goes wrong most\n\nThere is a pattern, and it is useful.\n\nFabrication clusters on things that are **specific and rare**. Names, numbers, dates, page references, quotations, small local details, anything about a niche subject, anything about a person who is not famous.\n\nIt is much rarer on things that are **general and repeated**. Water boiling at 100 degrees at sea level appears in the training text in essentially that form, over and over, so it comes back reliably.\n\nThe reason follows from the mechanism. A pattern that appeared ten thousand times is deeply worn into the dials. A specific study's page number appeared once, or never, and the model still has to produce something, so it produces something of the right shape.\n\nThe cruel part: the confident tone is identical in both cases. Text about page numbers is written confidently by humans, so text about page numbers comes out confidently.\n\n## What does not help\n\n**Asking are you sure.** You will get text about being sure. It has no separate access to its own reliability, so the question just prompts another round of likely-sounding continuation, often shaped by whether your tone suggested you wanted it to back down.\n\n**Asking for a source.** If no search tool is attached, a request for a source is a request for text that looks like a source.\n\n**Assuming it improves with importance.** It does not know which of your questions is the one that matters.\n\n## What does help\n\nAttaching real search or real documents helps a great deal, and this is what most serious products now do. It does not end the problem, because the model can still misread or overstate what it retrieved, but you get links you can open, and opening them is the point.\n\nBeyond that, a working rule: **the more specific and checkable a claim is, the more you have to check it.** Which is the opposite of the instinct. Specific claims feel more trustworthy. Here, specificity is the warning.\n\nThis is getting better and is not solved. Anyone who tells you the problem is fixed is selling something."},"check":{"question":"A model gives you two statements written in the same confident tone: that water boils at 100 degrees at sea level, and that a 2019 study by a named author found a 34 percent reduction, page 12. Which should worry you more, and why?","options":[{"text":"The study. Rare, specific details are where a plausible-shaped guess is most likely to have been invented, while the boiling point appeared in the training text in that same form countless times.","correct":true},{"text":"Neither more than the other. The tone is identical and the model cannot tell you when it is unsure, so both are equally suspect.","correct":false,"tempting":"It is true that the model gives you no confidence signal, so treating everything as equally doubtful feels like the rigorous move, but it throws away a genuinely useful pattern."},{"text":"The boiling point, because it is a scientific claim with conditions and exceptions that the model has flattened.","correct":false,"tempting":"Reaching for nuance sounds careful and intellectually honest, but it confuses a claim being simplified with a claim being fabricated."},{"text":"The study is the safer of the two, because citations come from indexed papers rather than from generated text.","correct":false,"tempting":"Citations look like retrieved records, with the visual furniture of a database entry, so people assume a lookup must have happened somewhere."}]},"prev":{"slug":"not-a-search-engine","title":"Why it is not a search engine, and what changes when you bolt one on","url":"https://addaly.com/learn/ai-actually-explained/not-a-search-engine"},"next":{"slug":"anatomy-of-a-hallucination","title":"Where invention concentrates, and why it never says \"I don't know\"","url":"https://addaly.com/learn/ai-actually-explained/anatomy-of-a-hallucination"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why it can be wrong and sound completely sure', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/confidently-wrong (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why it can be wrong and sound completely sure","courseTitle":"AI, Actually Explained","module":{"index":4,"slug":"why-it-goes-wrong","title":"Why it gets things wrong"},"answer":null,"keyIdea":"Nothing in the model checks anything, and invention clusters exactly where the details get specific.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"there-is-no-checking-step","runs":[{"text":"There is no checking step"}]},{"t":"p","runs":[{"text":"Here is the uncomfortable part of the last lesson, stated plainly."}]},{"t":"p","runs":[{"text":"Producing a true sentence and producing a false one use exactly the same machinery. The model assembles likely-sounding continuations. Most of the time, likely-sounding and true line up, because the text it was trained on was mostly written by people trying to say true things. When they come apart, nothing notices, because there is nothing whose job is to notice."}]},{"t":"p","runs":[{"text":"This is usually called hallucination. That word is a bit flattering. The model is not malfunctioning when it invents a citation. It is doing precisely what it always does."}]},{"t":"h2","id":"what-it-looks-like-in-practice","runs":[{"text":"What it looks like in practice"}]},{"t":"p","runs":[{"text":"In 2023 two lawyers in New York filed a court brief containing six cases that did not exist. A chatbot had produced them: plausible names, plausible courts, plausible years, plausible quotations. The lawyers were fined. The cases had the exact texture of real citations, because the model had absorbed the texture of thousands of real ones."}]},{"t":"p","runs":[{"text":"The same thing happens with a bus route that sounds right for a city it has read about, a customer helpline number with the correct number of digits, a section of a country’s tax code with a convincing number, a research paper attributed to a real author who never wrote it."}]},{"t":"h2","id":"where-it-goes-wrong-most","runs":[{"text":"Where it goes wrong most"}]},{"t":"p","runs":[{"text":"There is a pattern, and it is useful."}]},{"t":"p","runs":[{"text":"Fabrication clusters on things that are "},{"text":"specific and rare","b":true},{"text":". Names, numbers, dates, page references, quotations, small local details, anything about a niche subject, anything about a person who is not famous."}]},{"t":"p","runs":[{"text":"It is much rarer on things that are "},{"text":"general and repeated","b":true},{"text":". Water boiling at 100 degrees at sea level appears in the training text in essentially that form, over and over, so it comes back reliably."}]},{"t":"p","runs":[{"text":"The reason follows from the mechanism. A pattern that appeared ten thousand times is deeply worn into the dials. A specific study’s page number appeared once, or never, and the model still has to produce something, so it produces something of the right shape."}]},{"t":"p","runs":[{"text":"The cruel part: the confident tone is identical in both cases. Text about page numbers is written confidently by humans, so text about page numbers comes out confidently."}]},{"t":"h2","id":"what-does-not-help","runs":[{"text":"What does not help"}]},{"t":"p","runs":[{"text":"Asking are you sure.","b":true},{"text":" You will get text about being sure. It has no separate access to its own reliability, so the question just prompts another round of likely-sounding continuation, often shaped by whether your tone suggested you wanted it to back down."}]},{"t":"p","runs":[{"text":"Asking for a source.","b":true},{"text":" If no search tool is attached, a request for a source is a request for text that looks like a source."}]},{"t":"p","runs":[{"text":"Assuming it improves with importance.","b":true},{"text":" It does not know which of your questions is the one that matters."}]},{"t":"h2","id":"what-does-help","runs":[{"text":"What does help"}]},{"t":"p","runs":[{"text":"Attaching real search or real documents helps a great deal, and this is what most serious products now do. It does not end the problem, because the model can still misread or overstate what it retrieved, but you get links you can open, and opening them is the point."}]},{"t":"p","runs":[{"text":"Beyond that, a working rule: "},{"text":"the more specific and checkable a claim is, the more you have to check it.","b":true},{"text":" Which is the opposite of the instinct. Specific claims feel more trustworthy. Here, specificity is the warning."}]},{"t":"p","runs":[{"text":"This is getting better and is not solved. Anyone who tells you the problem is fixed is selling something."}]}],"check":{"question":[{"text":"A model gives you two statements written in the same confident tone: that water boils at 100 degrees at sea level, and that a 2019 study by a named author found a 34 percent reduction, page 12. Which should worry you more, and why?"}],"options":[{"text":[{"text":"The study. Rare, specific details are where a plausible-shaped guess is most likely to have been invented, while the boiling point appeared in the training text in that same form countless times."}],"correct":true,"tempting":null},{"text":[{"text":"Neither more than the other. The tone is identical and the model cannot tell you when it is unsure, so both are equally suspect."}],"correct":false,"tempting":[{"text":"It is true that the model gives you no confidence signal, so treating everything as equally doubtful feels like the rigorous move, but it throws away a genuinely useful pattern."}]},{"text":[{"text":"The boiling point, because it is a scientific claim with conditions and exceptions that the model has flattened."}],"correct":false,"tempting":[{"text":"Reaching for nuance sounds careful and intellectually honest, but it confuses a claim being simplified with a claim being fabricated."}]},{"text":[{"text":"The study is the safer of the two, because citations come from indexed papers rather than from generated text."}],"correct":false,"tempting":[{"text":"Citations look like retrieved records, with the visual furniture of a database entry, so people assume a lookup must have happened somewhere."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"not-a-search-engine","title":"Why it is not a search engine, and what changes when you bolt one on","minutes":9},"next":{"slug":"anatomy-of-a-hallucination","title":"Where invention concentrates, and why it never says “I don’t know”","minutes":9}}}