{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"retrieval-does-not-mean-true","title":"Retrieval, and its three failure points","url":"https://addaly.com/learn/ai-and-you/retrieval-does-not-mean-true","position":21,"of":73,"minutes":9,"answer":"Retrieval fetches the nearest chunks, not the sufficient or correct ones, so near-identical old versions win, gaps go unreported, and a working citation link is not evidence that the linked text supports the sentence.","keyIdea":"Retrieval fetches the nearest chunks, not the sufficient or correct ones, so near-identical old versions win, gaps go unreported, and a working citation link is not evidence that the linked text supports the sentence."},"body":{"format":"markdown","text":"## What \"it searched for that\" actually means\n\nMost serious AI products no longer answer from the model alone. They retrieve first. The pattern has a name (retrieval-augmented generation, usually shortened to RAG) and it is worth understanding at the level of the plumbing, because it is now behind most enterprise assistants, most \"chat with your documents\" tools, and the search summaries you meet daily.\n\nThe pipeline has four steps.\n\n1. **Chunking.** Your documents are cut into pieces, typically a few hundred words each.\n2. **Embedding.** Each chunk is converted into a list of numbers (an embedding) positioned so that pieces of text with similar meaning sit close together in that space.\n3. **Retrieval.** Your question is embedded the same way, and the chunks nearest to it are pulled out. Usually three to ten of them.\n4. **Generation.** Those chunks are pasted into the model's context along with your question and an instruction like \"answer using only the provided material\".\n\nThat instruction is a request, not a constraint. Nothing in the machinery enforces it.\n\n> Figure: Retrieval-augmented generation, with its three failure points. Steps: 1. Chunk (Documents cut into pieces of a few hundred words; a table's heading can land in one chunk and its numbers in another); 2. Embed (Each chunk becomes a list of numbers placed near chunks with similar meaning); 3. Retrieve the nearest chunks (Failure one: the 2019 policy scores as similar as the 2024 one. Failure two: nothing relevant exists and the nearest chunks come back anyway); 4. Generate, 'using only the material provided' (Failure three: the sentence is written fluently whether or not the retrieved passage supports it). The instruction at the last step is a request, not a constraint. A system failing at retrieval and a system failing at faithfulness need different fixes, and one accuracy number hides which you have.\n\n## Failure one: it retrieved the wrong thing\n\nSimilarity of meaning is not relevance. Ask about the 2024 refund policy and the retriever may return the 2019 refund policy, because the two chunks are nearly identical in meaning: that is precisely why they score as similar.\n\nThis is the single most common cause of confidently wrong answers from document assistants, and it produces a distinctive symptom: the answer is coherent, plausible, and describes a real policy that is not the current one. Anything where recency, version or jurisdiction matters is exposed.\n\nThe second version of this failure is the chunk boundary. Cut a document into 400-word pieces and a table's heading ends up in one chunk and its numbers in another. The retrieved piece is then technically correct and missing the thing that made it meaningful.\n\n## Failure two: nothing relevant existed and it answered anyway\n\nIf the answer is not in the corpus, the retriever still returns its best matches: retrieval returns the nearest chunks, not the sufficient ones. The model then has irrelevant material and a question, and its trained instinct is to be helpful.\n\nA well-built system checks the similarity score and refuses below a threshold. Most systems do not, because refusing looks worse in a demo than answering. The observable sign is an answer that cites documents which turn out to be about a neighbouring topic.\n\n## Failure three: the citation does not support the sentence\n\nThe system attaches a source link to each claim. The link works. The document is genuine. And the sentence overstates it, merges two chunks into a claim neither made, or attaches a number from one paragraph to a subject from another.\n\nStudies of commercial search assistants have repeatedly found meaningful rates of citations that do not support the statement attached to them. The presence of a footnote is a design element, not a verification.\n\n## What to do as a user\n\n**Click the citation and find the sentence.** If you cannot locate text supporting the claim, treat the claim as unsupported regardless of how reasonable it sounds.\n\n**Ask what it did not find.** \"Which parts of my question were not covered by the retrieved material?\" often produces a much more honest answer than the original question did, because you have given it permission to report a gap.\n\n**Watch for version and date.** In any corpus containing successive editions of the same document, ask explicitly for the date of the source and check it.\n\n**Give it the document directly when you can.** Pasting the actual contract in beats retrieval over a corpus containing that contract, because you have removed the step that can pick the wrong one. The privacy cost of doing so is the subject of the next module.\n\n## What to do as a builder\n\nSurface the retrieved chunks, not just the citation. Set a relevance floor and let the system say it does not know. Include document dates and versions in the chunk text itself, so the model can see them. And measure the two things separately: how often the right chunk was retrieved, and how often the answer was faithful to what was retrieved. A system that is failing at retrieval and a system that is failing at faithfulness need completely different fixes, and one aggregate accuracy number hides which you have."},"check":{"question":"A company assistant answers a question about parental leave with a coherent policy description and a link to a real internal document. Staff later find the entitlement was changed last year. What most likely happened?","options":[{"text":"The retriever fetched the old version, which reads almost the same as the new one does","correct":true},{"text":"The model invented the policy even though it had the right document in front of it","correct":false,"tempting":"Possible, but less likely once retrieval is in play. Pure invention would rarely match a real superseded policy so exactly."},{"text":"The document was split into chunks too coarsely, so the model saw too much text","correct":false,"tempting":"Chunk size does cause failures, but coarse chunks add context. They do not pick the wrong version of a document."},{"text":"The embedding model is out of date and misses terms that the new policy uses","correct":false,"tempting":"Misidentifies the mechanism. The failure is choosing between two similar documents, not failing to understand new words."}],"why":"Two versions of one policy look alike to a similarity search, so the outdated one can win. The link is real and the answer is stale."},"prev":{"slug":"calibration-and-confidence","title":"What its confidence is worth","url":"https://addaly.com/learn/ai-and-you/calibration-and-confidence"},"next":{"slug":"search-results-are-not-sources","title":"When a summary launders a source","url":"https://addaly.com/learn/ai-and-you/search-results-are-not-sources"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Retrieval, and its three failure points', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/retrieval-does-not-mean-true (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Retrieval, and its three failure points","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":3,"slug":"truth-and-verification","title":"Truth, sources and verification"},"answer":null,"keyIdea":"Retrieval fetches the nearest chunks, not the sufficient or correct ones, so near-identical old versions win, gaps go unreported, and a working citation link is not evidence that the linked text supports the sentence.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"The instruction at the last step is a request, not a constraint. A system failing at retrieval and a system failing at faithfulness need different fixes, and one accuracy number hides which you have.","steps":[{"label":"Chunk","detail":"Documents cut into pieces of a few hundred words; a table’s heading can land in one chunk and its numbers in another"},{"label":"Embed","detail":"Each chunk becomes a list of numbers placed near chunks with similar meaning"},{"label":"Retrieve the nearest chunks","detail":"Failure one: the 2019 policy scores as similar as the 2024 one. Failure two: nothing relevant exists and the nearest chunks come back anyway"},{"label":"Generate, ‘using only the material provided’","detail":"Failure three: the sentence is written fluently whether or not the retrieved passage supports it"}],"title":"Retrieval-augmented generation, with its three failure points","type":"flow"},"alt":"Figure: Retrieval-augmented generation, with its three failure points. Steps: 1. Chunk (Documents cut into pieces of a few hundred words; a table’s heading can land in one chunk and its numbers in another); 2. Embed (Each chunk becomes a list of numbers placed near chunks with similar meaning); 3. Retrieve the nearest chunks (Failure one: the 2019 policy scores as similar as the 2024 one. Failure two: nothing relevant exists and the nearest chunks come back anyway); 4. Generate, ‘using only the material provided’ (Failure three: the sentence is written fluently whether or not the retrieved passage supports it). The instruction at the last step is a request, not a constraint. A system failing at retrieval and a system failing at faithfulness need different fixes, and one accuracy number hides which you have."}},"blocks":[{"t":"h2","id":"what-it-searched-for-that-actually-means","runs":[{"text":"What “it searched for that” actually means"}]},{"t":"p","runs":[{"text":"Most serious AI products no longer answer from the model alone. They retrieve first. The pattern has a name (retrieval-augmented generation, usually shortened to RAG) and it is worth understanding at the level of the plumbing, because it is now behind most enterprise assistants, most “chat with your documents” tools, and the search summaries you meet daily."}]},{"t":"p","runs":[{"text":"The pipeline has four steps."}]},{"t":"ol","start":1,"items":[[{"text":"Chunking.","b":true},{"text":" Your documents are cut into pieces, typically a few hundred words each."}],[{"text":"Embedding.","b":true},{"text":" Each chunk is converted into a list of numbers (an embedding) positioned so that pieces of text with similar meaning sit close together in that space."}],[{"text":"Retrieval.","b":true},{"text":" Your question is embedded the same way, and the chunks nearest to it are pulled out. Usually three to ten of them."}],[{"text":"Generation.","b":true},{"text":" Those chunks are pasted into the model’s context along with your question and an instruction like “answer using only the provided material”."}]]},{"t":"p","runs":[{"text":"That instruction is a request, not a constraint. Nothing in the machinery enforces it."}]},{"t":"h2","id":"failure-one-it-retrieved-the-wrong-thing","runs":[{"text":"Failure one: it retrieved the wrong thing"}]},{"t":"p","runs":[{"text":"Similarity of meaning is not relevance. Ask about the 2024 refund policy and the retriever may return the 2019 refund policy, because the two chunks are nearly identical in meaning: that is precisely why they score as similar."}]},{"t":"p","runs":[{"text":"This is the single most common cause of confidently wrong answers from document assistants, and it produces a distinctive symptom: the answer is coherent, plausible, and describes a real policy that is not the current one. Anything where recency, version or jurisdiction matters is exposed."}]},{"t":"p","runs":[{"text":"The second version of this failure is the chunk boundary. Cut a document into 400-word pieces and a table’s heading ends up in one chunk and its numbers in another. The retrieved piece is then technically correct and missing the thing that made it meaningful."}]},{"t":"h2","id":"failure-two-nothing-relevant-existed-and-it-answered-anyway","runs":[{"text":"Failure two: nothing relevant existed and it answered anyway"}]},{"t":"p","runs":[{"text":"If the answer is not in the corpus, the retriever still returns its best matches: retrieval returns the nearest chunks, not the sufficient ones. The model then has irrelevant material and a question, and its trained instinct is to be helpful."}]},{"t":"p","runs":[{"text":"A well-built system checks the similarity score and refuses below a threshold. Most systems do not, because refusing looks worse in a demo than answering. The observable sign is an answer that cites documents which turn out to be about a neighbouring topic."}]},{"t":"h2","id":"failure-three-the-citation-does-not-support-the-sentence","runs":[{"text":"Failure three: the citation does not support the sentence"}]},{"t":"p","runs":[{"text":"The system attaches a source link to each claim. The link works. The document is genuine. And the sentence overstates it, merges two chunks into a claim neither made, or attaches a number from one paragraph to a subject from another."}]},{"t":"p","runs":[{"text":"Studies of commercial search assistants have repeatedly found meaningful rates of citations that do not support the statement attached to them. The presence of a footnote is a design element, not a verification."}]},{"t":"h2","id":"what-to-do-as-a-user","runs":[{"text":"What to do as a user"}]},{"t":"p","runs":[{"text":"Click the citation and find the sentence.","b":true},{"text":" If you cannot locate text supporting the claim, treat the claim as unsupported regardless of how reasonable it sounds."}]},{"t":"p","runs":[{"text":"Ask what it did not find.","b":true},{"text":" “Which parts of my question were not covered by the retrieved material?” often produces a much more honest answer than the original question did, because you have given it permission to report a gap."}]},{"t":"p","runs":[{"text":"Watch for version and date.","b":true},{"text":" In any corpus containing successive editions of the same document, ask explicitly for the date of the source and check it."}]},{"t":"p","runs":[{"text":"Give it the document directly when you can.","b":true},{"text":" Pasting the actual contract in beats retrieval over a corpus containing that contract, because you have removed the step that can pick the wrong one. The privacy cost of doing so is the subject of the next module."}]},{"t":"h2","id":"what-to-do-as-a-builder","runs":[{"text":"What to do as a builder"}]},{"t":"p","runs":[{"text":"Surface the retrieved chunks, not just the citation. Set a relevance floor and let the system say it does not know. Include document dates and versions in the chunk text itself, so the model can see them. And measure the two things separately: how often the right chunk was retrieved, and how often the answer was faithful to what was retrieved. A system that is failing at retrieval and a system that is failing at faithfulness need completely different fixes, and one aggregate accuracy number hides which you have."}]}],"check":{"question":[{"text":"A company assistant answers a question about parental leave with a coherent policy description and a link to a real internal document. Staff later find the entitlement was changed last year. What most likely happened?"}],"options":[{"text":[{"text":"The retriever fetched the old version, which reads almost the same as the new one does"}],"correct":true,"tempting":null},{"text":[{"text":"The model invented the policy even though it had the right document in front of it"}],"correct":false,"tempting":[{"text":"Possible, but less likely once retrieval is in play. Pure invention would rarely match a real superseded policy so exactly."}]},{"text":[{"text":"The document was split into chunks too coarsely, so the model saw too much text"}],"correct":false,"tempting":[{"text":"Chunk size does cause failures, but coarse chunks add context. They do not pick the wrong version of a document."}]},{"text":[{"text":"The embedding model is out of date and misses terms that the new policy uses"}],"correct":false,"tempting":[{"text":"Misidentifies the mechanism. The failure is choosing between two similar documents, not failing to understand new words."}]}],"order":[1,2,0,3],"why":[{"text":"Two versions of one policy look alike to a similarity search, so the outdated one can win. The link is real and the answer is stale."}]},"prev":{"slug":"calibration-and-confidence","title":"What its confidence is worth","minutes":8},"next":{"slug":"search-results-are-not-sources","title":"When a summary launders a source","minutes":7}}}