Why it cannot tell you where it got that

Provenance is not stored

Ask a model where a claim came from and you will get a source. Whether that source exists is a separate matter, and the reason has nothing to do with honesty.

During training, text passes through the model and the weights are nudged. The text is not kept. What survives is an enormous set of numbers in which the statistical regularities of billions of documents have been superimposed on each other, the way many exposures on one photographic plate produce an image that belongs to no single exposure. There is no index from a weight back to a document. There is no field recording that this pattern came from a 2017 paper and that one from a forum post.

So when you ask for a citation and the model is not connected to a search tool, it is not looking anything up. It is generating the most plausible-looking citation for a claim of that kind — a title of the right shape, an author whose name appears in that field, a journal that publishes such work, a year in the right range, a volume and page number in the right format. Every component is individually plausible. The combination is a new object.

This is why fabricated citations are so convincing and so common. They are being produced by exactly the machinery that produces good prose, applied to a highly patterned genre.

Sometimes it does remember, exactly

The picture above is the general case. There is an important exception: models do memorise some training text verbatim, and it can be extracted.

Memorisation increases with how often a passage appeared, how large the model is, and how distinctive the text is. Research teams have shown that a well-designed prompt can make production models emit long stretches of training data word for word, including personal information that appeared on the public web. Popular quotations, famous opening lines, standard licence texts, widely copied code and boilerplate legal clauses are frequently reproduced exactly.

So the honest statement is not "models never store text". It is that models store some text unpredictably, without any marker distinguishing the recalled from the invented, and you cannot tell which you have received by looking at it. A verbatim memorised quotation and a fabricated one arrive in the same font.

This exception matters for two reasons later in the course: it is the mechanism behind privacy leaks from training data, and it is one of the factual claims at issue in the copyright litigation.

What changes when a tool is attached

Many products now search the web, or a company's own documents, before answering. That is a real improvement and it changes the failure mode rather than removing it.

With retrieval, the citation usually points at a document that exists, because a document was actually fetched. What is not guaranteed is that the document says what the sentence claims. The generation step still writes the sentence, and it writes it fluently whether or not the retrieved passage supports it. You will meet this in detail two lessons from now.

The practical distinction: an unretrieved citation may not exist; a retrieved citation may not support the claim. Both need checking, in different ways. For the first, check existence. For the second, open the link and find the sentence.

Working practice

Four habits, each cheap.

Never accept a citation you have not opened. Not "searched for" — opened, and read the relevant part. A large fraction of fabricated references have real-sounding titles that return nothing, and a smaller, nastier fraction match a real paper about something else.

Ask for the claim and the source separately. Get the assertion first, then go and find support for it yourself. Asking the model for both at once invites it to construct a matched pair, which is the failure you are trying to detect.

Prefer identifiers you can resolve. A DOI, an ISBN, a case citation, a standard number. These either resolve or they do not, in one click, and they are harder to fake convincingly than a title.

Treat a quotation as the highest-risk object in any output. Quotation marks around invented text is the most damaging thing these systems do, because a quotation is what a reader trusts most and checks least. If you are going to quote, find the original.

The reframe

Stop thinking of the model as a source. It is not one, in the sense that a library or an archive is. It is a very capable writer with an unusually good memory for the shape of things and no memory at all for where it read them.

A writer like that is genuinely useful. You would still, before publishing under your own name, go and check the references.