{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"retrieval-that-helps","title":"Retrieval: what it fixes and what it does not","url":"https://addaly.com/learn/building-with-ai/retrieval-that-helps","position":32,"of":80,"minutes":8,"answer":"Retrieval decides what the model can know; when RAG fails, it is usually search that failed.","keyIdea":"Retrieval decides what the model can know; when RAG fails, it is usually search that failed."},"body":{"format":"markdown","text":"## The idea in one line\n\nRetrieval-augmented generation means: before you ask the model a question, go and find the relevant text yourself, paste it into the prompt, and tell the model to answer from it. That is all. There is no special model and no special API.\n\n```python\ndef answer(question):\n    chunks = search(question, k=6)                 # your search, whatever it is\n    context = \"\\n\\n\".join(\n        f\"[{i+1}] {c.title} (updated {c.updated})\\n{c.text}\"\n        for i, c in enumerate(chunks)\n    )\n    prompt = f\"\"\"Answer using only the passages below. Cite sources like [2].\nIf the passages do not contain the answer, say \"I don't know\" and name what is missing.\nIf two passages disagree, prefer the most recently updated one and say that you did.\n\nPassages:\n{context}\n\nQuestion: {question}\"\"\"\n    return call_model(prompt)\n```\n\n## What it fixes\n\n**Knowledge the model never had.** Your company's leave policy, a school's fee structure, last week's incident report. No amount of model quality helps here; the information is not in the weights.\n\n**Staleness.** A model trained a year ago has a year-old view of the world. Retrieval reads today's document.\n\n**Scale.** You cannot paste 40,000 support articles into a prompt, and you would not want to pay for it if you could.\n\n**Citations.** This is underrated. When the answer names its sources, a user can check it, and a wrong answer becomes a correctable one rather than a trust problem.\n\n## What it does not fix\n\n**It does not stop hallucination.** It reduces it when the right passage is found. It can make it worse when a confident, wrong or outdated passage is found, because now the model has evidence for the wrong answer.\n\n**It does not make the model reason better.** If the question needs a chain of inference across five documents, retrieval hands over five documents and the reasoning is still the hard part.\n\n**It does not answer aggregate questions.** \"How many refunds did we issue in Q3?\" is not a retrieval question. No single passage contains the answer, and pulling six passages that each mention a refund produces a confident, invented number. That question is SQL. Route it to SQL.\n\n**It does not fix a bad search index.** This is the big one.\n\n## Most RAG failures are search failures\n\nIf the passage containing the answer is not in the top k, nothing downstream can save you. Not a better model, not a cleverer prompt, not a longer context window. The answer was never in the room.\n\nSo measure the two halves separately. Take 50 real questions, note which document should answer each, and compute how often it appears in your top k. If retrieval recall is 60%, your ceiling is roughly 60% and prompt work is wasted effort.\n\n```python\nhits = sum(1 for q in golden if q.expected_doc in {c.doc_id for c in search(q.text, k=6)})\nprint(f\"recall@6: {hits}/{len(golden)}\")\n```\n\nA related and unglamorous truth: plain keyword search often beats a naive embedding index on the things users actually type. Product codes, error codes, invoice numbers, people's names, drug names. `ERR_4021` and `ERR_4012` sit close together in embedding space and are completely different problems. Run both and combine them.\n\n## Freshness is your job, not the model's\n\nA support bot in Jakarta answers policy questions well for months, then starts confidently stating a rule that changed last year. The logs show both the old and the new policy page were retrieved. The model had no way to tell which one was current: they both look like policy.\n\nNothing in the model fixes that. The index has to. Delete or archive superseded documents, stamp every chunk with an updated date, filter to current versions before searching, and put the date in the passage text so the model can see it and so your prompt rule about preferring recent sources has something to act on.\n\nRetrieval decides what the model is allowed to know. Spend your time there."},"check":{"question":"A university's fee-enquiry bot answers most questions well, but on refund deadlines it confidently states a rule that was replaced last year. The trace shows both the old and the new policy page were in the retrieved passages. What is the most useful reading of this failure?","options":[{"text":"The model hallucinated, so the fix is a stronger model.","correct":false,"tempting":"Any confident wrong answer looks like hallucination, and swapping models is the first lever most teams reach for."},{"text":"Retrieval put a superseded document in the context and the model has no way to tell which version is current; recency has to be handled in the index and the prompt.","correct":true},{"text":"The retrieval depth is too low; raising k so the new policy has more weight would fix it.","correct":false,"tempting":"More context intuitively feels like more truth, and k is the easiest knob to turn, even though the new policy was already retrieved."},{"text":"The model's knowledge is out of date, so it should be fine-tuned on the new policy.","correct":false,"tempting":"It presents as a knowledge problem, and training on the correct policy sounds like the direct route to fixing knowledge."}]},"prev":{"slug":"human-in-the-loop-handoff","title":"Approval gates, undo, and handing over to a person","url":"https://addaly.com/learn/building-with-ai/human-in-the-loop-handoff"},"next":{"slug":"embeddings-what-the-numbers-mean","title":"What an embedding is, and what \"close\" actually means","url":"https://addaly.com/learn/building-with-ai/embeddings-what-the-numbers-mean"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Retrieval: what it fixes and what it does not', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/retrieval-that-helps (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Retrieval: what it fixes and what it does not","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"Retrieval decides what the model can know; when RAG fails, it is usually search that failed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-idea-in-one-line","runs":[{"text":"The idea in one line"}]},{"t":"p","runs":[{"text":"Retrieval-augmented generation means: before you ask the model a question, go and find the relevant text yourself, paste it into the prompt, and tell the model to answer from it. That is all. There is no special model and no special API."}]},{"t":"code","lang":"python","label":"Python","text":"def answer(question):\n    chunks = search(question, k=6)                 # your search, whatever it is\n    context = \"\\n\\n\".join(\n        f\"[{i+1}] {c.title} (updated {c.updated})\\n{c.text}\"\n        for i, c in enumerate(chunks)\n    )\n    prompt = f\"\"\"Answer using only the passages below. Cite sources like [2].\nIf the passages do not contain the answer, say \"I don't know\" and name what is missing.\nIf two passages disagree, prefer the most recently updated one and say that you did.\n\nPassages:\n{context}\n\nQuestion: {question}\"\"\"\n    return call_model(prompt)"},{"t":"h2","id":"what-it-fixes","runs":[{"text":"What it fixes"}]},{"t":"p","runs":[{"text":"Knowledge the model never had.","b":true},{"text":" Your company’s leave policy, a school’s fee structure, last week’s incident report. No amount of model quality helps here; the information is not in the weights."}]},{"t":"p","runs":[{"text":"Staleness.","b":true},{"text":" A model trained a year ago has a year-old view of the world. Retrieval reads today’s document."}]},{"t":"p","runs":[{"text":"Scale.","b":true},{"text":" You cannot paste 40,000 support articles into a prompt, and you would not want to pay for it if you could."}]},{"t":"p","runs":[{"text":"Citations.","b":true},{"text":" This is underrated. When the answer names its sources, a user can check it, and a wrong answer becomes a correctable one rather than a trust problem."}]},{"t":"h2","id":"what-it-does-not-fix","runs":[{"text":"What it does not fix"}]},{"t":"p","runs":[{"text":"It does not stop hallucination.","b":true},{"text":" It reduces it when the right passage is found. It can make it worse when a confident, wrong or outdated passage is found, because now the model has evidence for the wrong answer."}]},{"t":"p","runs":[{"text":"It does not make the model reason better.","b":true},{"text":" If the question needs a chain of inference across five documents, retrieval hands over five documents and the reasoning is still the hard part."}]},{"t":"p","runs":[{"text":"It does not answer aggregate questions.","b":true},{"text":" “How many refunds did we issue in Q3?” is not a retrieval question. No single passage contains the answer, and pulling six passages that each mention a refund produces a confident, invented number. That question is SQL. Route it to SQL."}]},{"t":"p","runs":[{"text":"It does not fix a bad search index.","b":true},{"text":" This is the big one."}]},{"t":"h2","id":"most-rag-failures-are-search-failures","runs":[{"text":"Most RAG failures are search failures"}]},{"t":"p","runs":[{"text":"If the passage containing the answer is not in the top k, nothing downstream can save you. Not a better model, not a cleverer prompt, not a longer context window. The answer was never in the room."}]},{"t":"p","runs":[{"text":"So measure the two halves separately. Take 50 real questions, note which document should answer each, and compute how often it appears in your top k. If retrieval recall is 60%, your ceiling is roughly 60% and prompt work is wasted effort."}]},{"t":"code","lang":"python","label":"Python","text":"hits = sum(1 for q in golden if q.expected_doc in {c.doc_id for c in search(q.text, k=6)})\nprint(f\"recall@6: {hits}/{len(golden)}\")"},{"t":"p","runs":[{"text":"A related and unglamorous truth: plain keyword search often beats a naive embedding index on the things users actually type. Product codes, error codes, invoice numbers, people’s names, drug names. "},{"text":"ERR_4021","code":true},{"text":" and "},{"text":"ERR_4012","code":true},{"text":" sit close together in embedding space and are completely different problems. Run both and combine them."}]},{"t":"h2","id":"freshness-is-your-job-not-the-model-s","runs":[{"text":"Freshness is your job, not the model’s"}]},{"t":"p","runs":[{"text":"A support bot in Jakarta answers policy questions well for months, then starts confidently stating a rule that changed last year. The logs show both the old and the new policy page were retrieved. The model had no way to tell which one was current: they both look like policy."}]},{"t":"p","runs":[{"text":"Nothing in the model fixes that. The index has to. Delete or archive superseded documents, stamp every chunk with an updated date, filter to current versions before searching, and put the date in the passage text so the model can see it and so your prompt rule about preferring recent sources has something to act on."}]},{"t":"p","runs":[{"text":"Retrieval decides what the model is allowed to know. Spend your time there."}]}],"check":{"question":[{"text":"A university’s fee-enquiry bot answers most questions well, but on refund deadlines it confidently states a rule that was replaced last year. The trace shows both the old and the new policy page were in the retrieved passages. What is the most useful reading of this failure?"}],"options":[{"text":[{"text":"The model hallucinated, so the fix is a stronger model."}],"correct":false,"tempting":[{"text":"Any confident wrong answer looks like hallucination, and swapping models is the first lever most teams reach for."}]},{"text":[{"text":"Retrieval put a superseded document in the context and the model has no way to tell which version is current; recency has to be handled in the index and the prompt."}],"correct":true,"tempting":null},{"text":[{"text":"The retrieval depth is too low; raising k so the new policy has more weight would fix it."}],"correct":false,"tempting":[{"text":"More context intuitively feels like more truth, and k is the easiest knob to turn, even though the new policy was already retrieved."}]},{"text":[{"text":"The model’s knowledge is out of date, so it should be fine-tuned on the new policy."}],"correct":false,"tempting":[{"text":"It presents as a knowledge problem, and training on the correct policy sounds like the direct route to fixing knowledge."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"human-in-the-loop-handoff","title":"Approval gates, undo, and handing over to a person","minutes":8},"next":{"slug":"embeddings-what-the-numbers-mean","title":"What an embedding is, and what “close” actually means","minutes":10}}}