{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"the-model-cannot-cite-itself","title":"Why it cannot tell you where it got that","url":"https://addaly.com/learn/ai-and-you/the-model-cannot-cite-itself","position":19,"of":73,"minutes":9,"answer":"Training discards the documents and keeps only statistics, so an unretrieved citation is generated rather than recalled, and even when a model does reproduce memorised text exactly, nothing marks it as recalled rather than invented.","keyIdea":"Training discards the documents and keeps only statistics, so an unretrieved citation is generated rather than recalled, and even when a model does reproduce memorised text exactly, nothing marks it as recalled rather than invented."},"body":{"format":"markdown","text":"## Provenance is not stored\n\nAsk a model where a claim came from and you will get a source. Whether that source exists is a separate matter, and the reason has nothing to do with honesty.\n\nDuring training, text passes through the model and the weights are nudged. The text is not kept. What survives is an enormous set of numbers in which the statistical regularities of billions of documents have been superimposed on each other, the way many exposures on one photographic plate produce an image that belongs to no single exposure. There is no index from a weight back to a document. There is no field recording that this pattern came from a 2017 paper and that one from a forum post.\n\nSo when you ask for a citation and the model is not connected to a search tool, it is not looking anything up. It is generating the most plausible-looking citation for a claim of that kind: a title of the right shape, an author whose name appears in that field, a journal that publishes such work, a year in the right range, a volume and page number in the right format. Every component is individually plausible. The combination is a new object.\n\nThis is why fabricated citations are so convincing and so common. They are being produced by exactly the machinery that produces good prose, applied to a highly patterned genre.\n\n## Sometimes it does remember, exactly\n\nThere is an important exception: models do memorise some training text verbatim, and it can be extracted.\n\nMemorisation increases with how often a passage appeared, how large the model is, and how distinctive the text is. Research teams have shown that a well-designed prompt can make production models emit long stretches of training data word for word, including personal information that appeared on the public web. Popular quotations, famous opening lines, standard licence texts, widely copied code and boilerplate legal clauses are frequently reproduced exactly.\n\nSo the honest statement is not \"models never store text\". It is that models store some text unpredictably, without any marker distinguishing the recalled from the invented, and you cannot tell which you have received by looking at it. A verbatim memorised quotation and a fabricated one arrive in the same font.\n\nThis exception matters for two reasons later in the course: it is the mechanism behind privacy leaks from training data, and it is one of the factual claims at issue in the copyright litigation.\n\n## What changes when a tool is attached\n\nMany products now search the web, or a company's own documents, before answering. That is a real improvement and it changes the failure mode rather than removing it.\n\nWith retrieval, the citation usually points at a document that exists, because a document was actually fetched. What is not guaranteed is that the document says what the sentence claims. The generation step still writes the sentence, and it writes it fluently whether or not the retrieved passage supports it. You will meet this in detail in [Retrieval, and its three failure points](/learn/ai-and-you/retrieval-does-not-mean-true).\n\nThe practical distinction: **an unretrieved citation may not exist; a retrieved citation may not support the claim.** Both need checking, in different ways. For the first, check existence. For the second, open the link and find the sentence.\n\n> Figure: Two kinds of citation, two different checks. Unretrieved: the model wrote it from memory: May not exist at all; Every part plausible: title shape, author in the field, journal, year, page numbers; Check: does it exist? Search the exact title; Prefer a DOI, ISBN or case citation that resolves in one click. Retrieved: a search step fetched it: The document usually exists; The sentence may overstate it, merge two chunks, or misattach a number; Check: open the link and find the sentence; Watch the version and the date of the source. Both need checking. The first can be a new object assembled from the shape of citations; the second points at a real document that may not say what the sentence claims.\n\n## Working practice\n\n**Never accept a citation you have not opened.** Not \"searched for\": opened, and read the relevant part. A large fraction of fabricated references have real-sounding titles that return nothing, and a smaller, nastier fraction match a real paper about something else.\n\n**Ask for the claim and the source separately.** Get the assertion first, then go and find support for it yourself. Asking the model for both at once invites it to construct a matched pair, which is the failure you are trying to detect.\n\n**Prefer identifiers you can resolve.** A DOI, an ISBN, a case citation, a standard number. These either resolve or they do not, in one click, and they are harder to fake convincingly than a title.\n\n**Treat a quotation as the highest-risk object in any output.** Quotation marks around invented text is the most damaging thing these systems do, because a quotation is what a reader trusts most and checks least. If you are going to quote, find the original.\n\n## The reframe\n\nStop thinking of the model as a source. It is not one, in the sense that a library or an archive is. It is a very capable writer with an unusually good memory for the shape of things and no memory at all for where it read them."},"check":{"question":"A model with web search attached returns a claim with a link to a real, working article. What still needs checking, and why?","options":[{"text":"Nothing, since the retrieval step shows the source exists and so supports the claim","correct":false,"tempting":"Treats existence as support. Retrieval settles whether the document is real, not whether the sentence reflects it."},{"text":"Whether the model's stated confidence in the claim is high enough to justify the citation","correct":false,"tempting":"Uses stated confidence as a gate, but confidence comes from the same process as the text and does not track support."},{"text":"Whether the article really contains the claim, since the sentence was still generated","correct":true},{"text":"Whether the article was in its training data, since memorised text is more reliable","correct":false,"tempting":"Inverts the order of reliability. Memorised and invented output look the same on the page."}],"why":"A real link proves the page exists. Only reading it shows whether the sentence built on it is true to it."},"prev":{"slug":"hallucination-and-verification","title":"Hallucination, and how to check","url":"https://addaly.com/learn/ai-and-you/hallucination-and-verification"},"next":{"slug":"calibration-and-confidence","title":"What its confidence is worth","url":"https://addaly.com/learn/ai-and-you/calibration-and-confidence"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why it cannot tell you where it got that', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/the-model-cannot-cite-itself (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why it cannot tell you where it got that","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":3,"slug":"truth-and-verification","title":"Truth, sources and verification"},"answer":null,"keyIdea":"Training discards the documents and keeps only statistics, so an unretrieved citation is generated rather than recalled, and even when a model does reproduce memorised text exactly, nothing marks it as recalled rather than invented.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Both need checking. The first can be a new object assembled from the shape of citations; the second points at a real document that may not say what the sentence claims.","left":{"title":"Unretrieved: the model wrote it from memory","items":["May not exist at all","Every part plausible: title shape, author in the field, journal, year, page numbers","Check: does it exist? Search the exact title","Prefer a DOI, ISBN or case citation that resolves in one click"]},"right":{"title":"Retrieved: a search step fetched it","items":["The document usually exists","The sentence may overstate it, merge two chunks, or misattach a number","Check: open the link and find the sentence","Watch the version and the date of the source"]},"title":"Two kinds of citation, two different checks","type":"compare"},"alt":"Figure: Two kinds of citation, two different checks. Unretrieved: the model wrote it from memory: May not exist at all; Every part plausible: title shape, author in the field, journal, year, page numbers; Check: does it exist? Search the exact title; Prefer a DOI, ISBN or case citation that resolves in one click. Retrieved: a search step fetched it: The document usually exists; The sentence may overstate it, merge two chunks, or misattach a number; Check: open the link and find the sentence; Watch the version and the date of the source. Both need checking. The first can be a new object assembled from the shape of citations; the second points at a real document that may not say what the sentence claims."}},"blocks":[{"t":"h2","id":"provenance-is-not-stored","runs":[{"text":"Provenance is not stored"}]},{"t":"p","runs":[{"text":"Ask a model where a claim came from and you will get a source. Whether that source exists is a separate matter, and the reason has nothing to do with honesty."}]},{"t":"p","runs":[{"text":"During training, text passes through the model and the weights are nudged. The text is not kept. What survives is an enormous set of numbers in which the statistical regularities of billions of documents have been superimposed on each other, the way many exposures on one photographic plate produce an image that belongs to no single exposure. There is no index from a weight back to a document. There is no field recording that this pattern came from a 2017 paper and that one from a forum post."}]},{"t":"p","runs":[{"text":"So when you ask for a citation and the model is not connected to a search tool, it is not looking anything up. It is generating the most plausible-looking citation for a claim of that kind: a title of the right shape, an author whose name appears in that field, a journal that publishes such work, a year in the right range, a volume and page number in the right format. Every component is individually plausible. The combination is a new object."}]},{"t":"p","runs":[{"text":"This is why fabricated citations are so convincing and so common. They are being produced by exactly the machinery that produces good prose, applied to a highly patterned genre."}]},{"t":"h2","id":"sometimes-it-does-remember-exactly","runs":[{"text":"Sometimes it does remember, exactly"}]},{"t":"p","runs":[{"text":"There is an important exception: models do memorise some training text verbatim, and it can be extracted."}]},{"t":"p","runs":[{"text":"Memorisation increases with how often a passage appeared, how large the model is, and how distinctive the text is. Research teams have shown that a well-designed prompt can make production models emit long stretches of training data word for word, including personal information that appeared on the public web. Popular quotations, famous opening lines, standard licence texts, widely copied code and boilerplate legal clauses are frequently reproduced exactly."}]},{"t":"p","runs":[{"text":"So the honest statement is not “models never store text”. It is that models store some text unpredictably, without any marker distinguishing the recalled from the invented, and you cannot tell which you have received by looking at it. A verbatim memorised quotation and a fabricated one arrive in the same font."}]},{"t":"p","runs":[{"text":"This exception matters for two reasons later in the course: it is the mechanism behind privacy leaks from training data, and it is one of the factual claims at issue in the copyright litigation."}]},{"t":"h2","id":"what-changes-when-a-tool-is-attached","runs":[{"text":"What changes when a tool is attached"}]},{"t":"p","runs":[{"text":"Many products now search the web, or a company’s own documents, before answering. That is a real improvement and it changes the failure mode rather than removing it."}]},{"t":"p","runs":[{"text":"With retrieval, the citation usually points at a document that exists, because a document was actually fetched. What is not guaranteed is that the document says what the sentence claims. The generation step still writes the sentence, and it writes it fluently whether or not the retrieved passage supports it. You will meet this in detail in "},{"text":"Retrieval, and its three failure points","href":"https://addaly.com/learn/ai-and-you/retrieval-does-not-mean-true"},{"text":"."}]},{"t":"p","runs":[{"text":"The practical distinction: "},{"text":"an unretrieved citation may not exist; a retrieved citation may not support the claim.","b":true},{"text":" Both need checking, in different ways. For the first, check existence. For the second, open the link and find the sentence."}]},{"t":"h2","id":"working-practice","runs":[{"text":"Working practice"}]},{"t":"p","runs":[{"text":"Never accept a citation you have not opened.","b":true},{"text":" Not “searched for”: opened, and read the relevant part. A large fraction of fabricated references have real-sounding titles that return nothing, and a smaller, nastier fraction match a real paper about something else."}]},{"t":"p","runs":[{"text":"Ask for the claim and the source separately.","b":true},{"text":" Get the assertion first, then go and find support for it yourself. Asking the model for both at once invites it to construct a matched pair, which is the failure you are trying to detect."}]},{"t":"p","runs":[{"text":"Prefer identifiers you can resolve.","b":true},{"text":" A DOI, an ISBN, a case citation, a standard number. These either resolve or they do not, in one click, and they are harder to fake convincingly than a title."}]},{"t":"p","runs":[{"text":"Treat a quotation as the highest-risk object in any output.","b":true},{"text":" Quotation marks around invented text is the most damaging thing these systems do, because a quotation is what a reader trusts most and checks least. If you are going to quote, find the original."}]},{"t":"h2","id":"the-reframe","runs":[{"text":"The reframe"}]},{"t":"p","runs":[{"text":"Stop thinking of the model as a source. It is not one, in the sense that a library or an archive is. It is a very capable writer with an unusually good memory for the shape of things and no memory at all for where it read them."}]}],"check":{"question":[{"text":"A model with web search attached returns a claim with a link to a real, working article. What still needs checking, and why?"}],"options":[{"text":[{"text":"Nothing, since the retrieval step shows the source exists and so supports the claim"}],"correct":false,"tempting":[{"text":"Treats existence as support. Retrieval settles whether the document is real, not whether the sentence reflects it."}]},{"text":[{"text":"Whether the model’s stated confidence in the claim is high enough to justify the citation"}],"correct":false,"tempting":[{"text":"Uses stated confidence as a gate, but confidence comes from the same process as the text and does not track support."}]},{"text":[{"text":"Whether the article really contains the claim, since the sentence was still generated"}],"correct":true,"tempting":null},{"text":[{"text":"Whether the article was in its training data, since memorised text is more reliable"}],"correct":false,"tempting":[{"text":"Inverts the order of reliability. Memorised and invented output look the same on the page."}]}],"order":[3,2,0,1],"why":[{"text":"A real link proves the page exists. Only reading it shows whether the sentence built on it is true to it."}]},"prev":{"slug":"hallucination-and-verification","title":"Hallucination, and how to check","minutes":8},"next":{"slug":"calibration-and-confidence","title":"What its confidence is worth","minutes":8}}}