AI, Safety and What Goes WrongLesson 19 of 73

Module 3 · Truth, sources and verification

Why it cannot tell you where it got that

In this lesson

  • 9 min read
  • 6 reading sections
  • Knowledge check
  • Lesson 2 of 8 in this chapter

This chapter’s goalEstimate from the shape of a question how likely a fabrication is, verify a claim at a cost proportional to the consequence of being wrong, and explain why a cited source that exists is not yet a verified claim.

This chapter’s media

The film explains chapter ideas. The audiobook narrates the complete teaching text.

FormatEnglishहिंदी
Chapter filmNot availableWatch2:37
Full audiobookNot availableNot available
Two kinds of citation, two different checksUnretrieved: the model wrote it from memoryMay not exist at allEvery part plausible: title shape, author inthe field, journal, year, page numbersCheck: does it exist? Search the exact titlePrefer a DOI, ISBN or case citation thatresolves in one clickRetrieved: a search step fetched itThe document usually existsThe sentence may overstate it, merge twochunks, or misattach a numberCheck: open the link and find the sentenceWatch the version and the date of the sourceBoth need checking. The first can be a new object assembled from the shape of citations; the secondpoints at a real document that may not say what the sentence claims.
Both need checking. The first can be a new object assembled from the shape of citations; the second points at a real document that may not say what the sentence claims.

Figure

A little at a time

Explore an idea, try it, then check what stays.

Step 1 of 8Free · no account needed
Lesson steps

Provenance is not stored

Ask a model where a claim came from and you will get a source. Whether that source exists is a separate matter, and the reason has nothing to do with honesty.

During training, text passes through the model and the weights are nudged. The text is not kept. What survives is an enormous set of numbers in which the statistical regularities of billions of documents have been superimposed on each other, the way many exposures on one photographic plate produce an image that belongs to no single exposure. There is no index from a weight back to a document. There is no field recording that this pattern came from a 2017 paper and that one from a forum post.

Your place and first choices stay on this browser. Classroom credit is recorded separately.

Stuck on something?Ask Rukmini, your AI tutor

How to cite: Addaly (2026) ‘Why it cannot tell you where it got that’, in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/the-model-cannot-cite-itself (Accessed: 21 November 2026).