{"course":{"slug":"ai-for-students","title":"AI for Students","url":"https://addaly.com/learn/ai-for-students","level":"beginner","track":"work"},"lesson":{"slug":"why-it-invents-things","title":"Why it invents things, and when","url":"https://addaly.com/learn/ai-for-students/why-it-invents-things","position":4,"of":73,"minutes":9,"answer":"Fabrication is the model doing its job (producing plausible text) so risk concentrates in names, numbers, dates and references, not in the general explanation around them.","keyIdea":"Fabrication is the model doing its job (producing plausible text) so risk concentrates in names, numbers, dates and references, not in the general explanation around them."},"body":{"format":"markdown","text":"## Fabrication is the system working normally\n\nA model that produces a fake reference is not malfunctioning. It is doing exactly what it was trained to do: produce the most plausible continuation. The most plausible continuation of *a citation for the claim that spaced repetition improves retention is* is a citation-shaped string. Author surname, initials, a year, a journal that publishes that sort of work, a plausible volume and page range. Every part is drawn from a well-learned pattern. The combination has never existed.\n\nThis is the single most useful thing to understand about these tools, because it tells you exactly where the risk lives. The risk is not in the reasoning. It is in the specifics.\n\n## The shape of the risk\n\nRank these by how likely the model is to be wrong, and the ranking is stable across every model you will use:\n\n1. **Very safe.** General explanations of well-established material. What photosynthesis does. Why a for-loop is slower than vectorised code. The structure of a five-paragraph argument.\n2. **Usually safe.** Standard worked methods. How to integrate by parts. How to balance a redox equation. These are procedures repeated thousands of times in the training text.\n3. **Risky.** Anything with a number, a name, a date, or a page reference attached. Population of a city in a particular year. Which scholar first argued a position. The exact wording of a statute.\n4. **Very risky.** Anything that must exist in the world for your claim to hold: a paper, a URL, a quotation, a court case, a library function, a command-line flag.\n\nCategories 3 and 4 are where the failures live, and they are also the parts a marker checks.\n\n> Figure: Where the risk sits inside one answer. Layers, top to bottom: Very safe: general explanation of well-established material (What photosynthesis does. Why a loop is slower than vectorised code.); Usually safe: standard worked methods (Integration by parts. Balancing a redox equation. Repeated thousands of times in the training text.); Risky: anything with a number, a name, a date or a page attached (A city's population in a given year. Who first argued a position. The wording of a statute.); Very risky: anything that must exist in the world for your claim to hold (A paper, a URL, a quotation, a court case, a library function, a command-line flag.). One paragraph usually contains all four layers at once, written in the same confident register, because there is no confidence dial connected to the output text. Underlining every proper noun, number, date and reference marks the bottom two layers in about ten seconds, and those are the parts a marker checks.\n\n## Why it sounds certain\n\nThere is no confidence dial connected to the output text. The model produces the tone that fits the context, and academic-sounding questions have academic-sounding answers in the training data, which are written confidently. A hedge like *I am not sure, but* appears only where hedges appeared in similar text, not where the model's internal probabilities were low.\n\nInternally, models do carry something like uncertainty: the probability assigned to each token. Research on this has a name, **calibration**, and models are moderately calibrated on multiple-choice questions and considerably worse on open text. But you cannot see those numbers in a chat window, and the model does not read them back to itself before writing. So confident phrasing carries almost no information about correctness.\n\n> Asking *are you sure?* is not a check. It is another prompt, and the most plausible continuation of a challenge is often an apology and a different answer, whether or not the first one was wrong.\n\n## The two questions that do work\n\nThe useless move is to ask the model to grade its own answer. Two things do help.\n\n**Ask it to produce something checkable.** Not *is this right* but *give me the steps*, *give me the DOI*, *write code that computes this*. A step, a DOI and a program can each be tested against something outside the conversation.\n\n**Ask it twice, cold.** Open a new chat. Rephrase the question so it does not echo the first answer. Genuine general knowledge is stable across rephrasings; fabrication usually is not. If you get two different citations for the same claim, at least one is invented and probably both are.\n\n## What retrieval does and does not fix\n\nMany tools now search the web and attach links. This helps, and it changes the failure rather than removing it. Now the model is summarising retrieved pages, so the summary can misstate a page that is itself correct, and the link can point at a real page that does not contain the claim. Studies of search-grounded assistants keep finding the same pattern: the citations exist, and a meaningful fraction do not support the sentence they are attached to.\n\nThe rule survives the upgrade. **Open the link.** If you did not open it, you did not check it.\n\n## Doing this without paying for anything\n\nEverything above works on free tiers. For verification specifically, the free tools that matter are Google Scholar, the DOI resolver at `doi.org`, arXiv, PubMed, and Unpaywall's browser extension, which finds legal free copies of paywalled papers. Your library almost certainly has an institutional login too, and the librarian is the most underused expert on your campus.\n\n## The habit to build\n\nWhen you read a model's answer, mentally underline every proper noun, number and reference. Those are the parts that need a source. The prose around them is usually fine. This takes about ten seconds and catches nearly everything that would otherwise cost you marks."},"check":{"question":"A student gets a model's answer, replies \"are you certain about that reference?\", and the model confirms it. What has the confirmation established?","options":[{"text":"That the reference is probably real, since models are trained to admit uncertainty when they have it.","correct":false,"tempting":"Assumes the hedging in training text tracks the model's internal probabilities; it tracks the surrounding style instead."},{"text":"Almost nothing, because the follow-up is just another prompt and the model has no lookup to consult.","correct":true},{"text":"That the reference is real if the model gave the same answer both times, since consistency implies retrieval from memory.","correct":false,"tempting":"Consistency within one conversation is expected: the earlier answer is in the context and heavily conditions what follows. Only a cold rephrasing in a fresh chat tests stability."},{"text":"That the model's confidence calibration was high for this answer, which is a reasonable proxy for accuracy on citations.","correct":false,"tempting":"Borrows a real concept, calibration, but confuses measured token probabilities with the confident tone of the output text; the chat window never shows the former."}]},"prev":{"slug":"what-a-model-actually-does","title":"What it is actually doing","url":"https://addaly.com/learn/ai-for-students/what-a-model-actually-does"},"next":{"slug":"the-fluency-trap","title":"The fluency trap","url":"https://addaly.com/learn/ai-for-students/the-fluency-trap"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why it invents things, and when', in AI for Students. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-students/why-it-invents-things (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why it invents things, and when","courseTitle":"AI for Students","module":{"index":1,"slug":"what-the-machine-is-doing","title":"What the machine is doing when it helps you"},"answer":null,"keyIdea":"Fabrication is the model doing its job (producing plausible text) so risk concentrates in names, numbers, dates and references, not in the general explanation around them.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Where the risk sits inside one answer","layers":[{"label":"Very safe: general explanation of well-established material","detail":"What photosynthesis does. Why a loop is slower than vectorised code."},{"label":"Usually safe: standard worked methods","detail":"Integration by parts. Balancing a redox equation. Repeated thousands of times in the training text."},{"label":"Risky: anything with a number, a name, a date or a page attached","detail":"A city’s population in a given year. Who first argued a position. The wording of a statute."},{"label":"Very risky: anything that must exist in the world for your claim to hold","detail":"A paper, a URL, a quotation, a court case, a library function, a command-line flag."}],"caption":"One paragraph usually contains all four layers at once, written in the same confident register, because there is no confidence dial connected to the output text. Underlining every proper noun, number, date and reference marks the bottom two layers in about ten seconds, and those are the parts a marker checks."},"alt":"Figure: Where the risk sits inside one answer. Layers, top to bottom: Very safe: general explanation of well-established material (What photosynthesis does. Why a loop is slower than vectorised code.); Usually safe: standard worked methods (Integration by parts. Balancing a redox equation. Repeated thousands of times in the training text.); Risky: anything with a number, a name, a date or a page attached (A city’s population in a given year. Who first argued a position. The wording of a statute.); Very risky: anything that must exist in the world for your claim to hold (A paper, a URL, a quotation, a court case, a library function, a command-line flag.). One paragraph usually contains all four layers at once, written in the same confident register, because there is no confidence dial connected to the output text. Underlining every proper noun, number, date and reference marks the bottom two layers in about ten seconds, and those are the parts a marker checks."}},"blocks":[{"t":"h2","id":"fabrication-is-the-system-working-normally","runs":[{"text":"Fabrication is the system working normally"}]},{"t":"p","runs":[{"text":"A model that produces a fake reference is not malfunctioning. It is doing exactly what it was trained to do: produce the most plausible continuation. The most plausible continuation of "},{"text":"a citation for the claim that spaced repetition improves retention is","i":true},{"text":" is a citation-shaped string. Author surname, initials, a year, a journal that publishes that sort of work, a plausible volume and page range. Every part is drawn from a well-learned pattern. The combination has never existed."}]},{"t":"p","runs":[{"text":"This is the single most useful thing to understand about these tools, because it tells you exactly where the risk lives. The risk is not in the reasoning. It is in the specifics."}]},{"t":"h2","id":"the-shape-of-the-risk","runs":[{"text":"The shape of the risk"}]},{"t":"p","runs":[{"text":"Rank these by how likely the model is to be wrong, and the ranking is stable across every model you will use:"}]},{"t":"ol","start":1,"items":[[{"text":"Very safe.","b":true},{"text":" General explanations of well-established material. What photosynthesis does. Why a for-loop is slower than vectorised code. The structure of a five-paragraph argument."}],[{"text":"Usually safe.","b":true},{"text":" Standard worked methods. How to integrate by parts. How to balance a redox equation. These are procedures repeated thousands of times in the training text."}],[{"text":"Risky.","b":true},{"text":" Anything with a number, a name, a date, or a page reference attached. Population of a city in a particular year. Which scholar first argued a position. The exact wording of a statute."}],[{"text":"Very risky.","b":true},{"text":" Anything that must exist in the world for your claim to hold: a paper, a URL, a quotation, a court case, a library function, a command-line flag."}]]},{"t":"p","runs":[{"text":"Categories 3 and 4 are where the failures live, and they are also the parts a marker checks."}]},{"t":"h2","id":"why-it-sounds-certain","runs":[{"text":"Why it sounds certain"}]},{"t":"p","runs":[{"text":"There is no confidence dial connected to the output text. The model produces the tone that fits the context, and academic-sounding questions have academic-sounding answers in the training data, which are written confidently. A hedge like "},{"text":"I am not sure, but","i":true},{"text":" appears only where hedges appeared in similar text, not where the model’s internal probabilities were low."}]},{"t":"p","runs":[{"text":"Internally, models do carry something like uncertainty: the probability assigned to each token. Research on this has a name, "},{"text":"calibration","b":true},{"text":", and models are moderately calibrated on multiple-choice questions and considerably worse on open text. But you cannot see those numbers in a chat window, and the model does not read them back to itself before writing. So confident phrasing carries almost no information about correctness."}]},{"t":"quote","runs":[{"text":"Asking "},{"text":"are you sure?","i":true},{"text":" is not a check. It is another prompt, and the most plausible continuation of a challenge is often an apology and a different answer, whether or not the first one was wrong."}]},{"t":"h2","id":"the-two-questions-that-do-work","runs":[{"text":"The two questions that do work"}]},{"t":"p","runs":[{"text":"The useless move is to ask the model to grade its own answer. Two things do help."}]},{"t":"p","runs":[{"text":"Ask it to produce something checkable.","b":true},{"text":" Not "},{"text":"is this right","i":true},{"text":" but "},{"text":"give me the steps","i":true},{"text":", "},{"text":"give me the DOI","i":true},{"text":", "},{"text":"write code that computes this","i":true},{"text":". A step, a DOI and a program can each be tested against something outside the conversation."}]},{"t":"p","runs":[{"text":"Ask it twice, cold.","b":true},{"text":" Open a new chat. Rephrase the question so it does not echo the first answer. Genuine general knowledge is stable across rephrasings; fabrication usually is not. If you get two different citations for the same claim, at least one is invented and probably both are."}]},{"t":"h2","id":"what-retrieval-does-and-does-not-fix","runs":[{"text":"What retrieval does and does not fix"}]},{"t":"p","runs":[{"text":"Many tools now search the web and attach links. This helps, and it changes the failure rather than removing it. Now the model is summarising retrieved pages, so the summary can misstate a page that is itself correct, and the link can point at a real page that does not contain the claim. Studies of search-grounded assistants keep finding the same pattern: the citations exist, and a meaningful fraction do not support the sentence they are attached to."}]},{"t":"p","runs":[{"text":"The rule survives the upgrade. "},{"text":"Open the link.","b":true},{"text":" If you did not open it, you did not check it."}]},{"t":"h2","id":"doing-this-without-paying-for-anything","runs":[{"text":"Doing this without paying for anything"}]},{"t":"p","runs":[{"text":"Everything above works on free tiers. For verification specifically, the free tools that matter are Google Scholar, the DOI resolver at "},{"text":"doi.org","code":true},{"text":", arXiv, PubMed, and Unpaywall’s browser extension, which finds legal free copies of paywalled papers. Your library almost certainly has an institutional login too, and the librarian is the most underused expert on your campus."}]},{"t":"h2","id":"the-habit-to-build","runs":[{"text":"The habit to build"}]},{"t":"p","runs":[{"text":"When you read a model’s answer, mentally underline every proper noun, number and reference. Those are the parts that need a source. The prose around them is usually fine. This takes about ten seconds and catches nearly everything that would otherwise cost you marks."}]}],"check":{"question":[{"text":"A student gets a model’s answer, replies “are you certain about that reference?”, and the model confirms it. What has the confirmation established?"}],"options":[{"text":[{"text":"That the reference is probably real, since models are trained to admit uncertainty when they have it."}],"correct":false,"tempting":[{"text":"Assumes the hedging in training text tracks the model’s internal probabilities; it tracks the surrounding style instead."}]},{"text":[{"text":"Almost nothing, because the follow-up is just another prompt and the model has no lookup to consult."}],"correct":true,"tempting":null},{"text":[{"text":"That the reference is real if the model gave the same answer both times, since consistency implies retrieval from memory."}],"correct":false,"tempting":[{"text":"Consistency within one conversation is expected: the earlier answer is in the context and heavily conditions what follows. Only a cold rephrasing in a fresh chat tests stability."}]},{"text":[{"text":"That the model’s confidence calibration was high for this answer, which is a reasonable proxy for accuracy on citations."}],"correct":false,"tempting":[{"text":"Borrows a real concept, calibration, but confuses measured token probabilities with the confident tone of the output text; the chat window never shows the former."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"what-a-model-actually-does","title":"What it is actually doing","minutes":9},"next":{"slug":"the-fluency-trap","title":"The fluency trap","minutes":8}}}