{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"retrieval-mechanically","title":"Retrieval, as a pipeline that can break in six places","url":"https://addaly.com/learn/how-llms-work/retrieval-mechanically","position":34,"of":78,"minutes":9,"answer":"Retrieval-augmented generation is a search system with a model on the end, and most of its failures are search failures that get blamed on the model.","keyIdea":"Retrieval-augmented generation is a search system with a model on the end, and most of its failures are search failures that get blamed on the model."},"body":{"format":"markdown","text":"## The pipeline, stated plainly\n\nRetrieval-augmented generation has an aura of sophistication that it does not deserve. It is this:\n\n1. **Split** your documents into chunks.\n2. **Embed** each chunk and store the vectors.\n3. **Embed** the incoming question.\n4. **Retrieve** the nearest chunks.\n5. **Paste** them into the prompt with an instruction to answer from them.\n6. **Generate.**\n\nNothing more. When it works well it is because steps 1 to 4 worked well. When it fails, the failure is usually in steps 1 to 4 and gets described as \"the model hallucinated\".\n\n> Figure: Six places retrieval breaks, in order of frequency. Steps: 1. Split the documents into chunks (A boundary through a table or an answer caps everything downstream); 2. Embed and store the chunks (Vectors from one provider's model are meaningless to another's); 3. Embed the question (\"The drug is safe\" and \"the drug is not safe\" sit very close together); 4. Retrieve the nearest few (If the right chunk is in the top five only 60% of the time, nothing above 60% is available); 5. Paste them into the prompt (Nine plausible distractors around one correct passage measurably lower accuracy); 6. Generate (The model may answer from its weights instead, or report an outdated policy faithfully and with a citation). Five of the six failures happen before the model is asked anything, and all six get reported as \"the model hallucinated\". Measure the stages apart: hand the model the correct chunk yourself, and whatever still fails is generation.\n\n## Why it exists\n\nModule one established that facts live in weights, set at training time, and cannot be edited by prompting. Retrieval sidesteps that. Instead of changing what the model knows, you change what it can see. This gives you:\n\n- **Current information** without retraining.\n- **Private information** the model was never trained on.\n- **Citations**, because you know which chunk each answer came from.\n- **Revocation**, because deleting a document removes it from future answers, which fine-tuning cannot do.\n\nThat last property is underrated and often decisive for anything with a compliance requirement.\n\n## Where it breaks, in order of frequency\n\n**The answer was never retrieved.** The model cannot use what it never received. Measure this separately from everything else: for a set of questions with known source documents, what fraction of the time is the right chunk in the top *k*? If that number is 60 per cent, no prompt engineering will get you above 60 per cent, and you are working on the wrong problem.\n\n**The chunk was retrieved but incomplete.** The answer spans a boundary; a table's header is in one chunk and its rows in another; a pronoun in the chunk refers to a subject named two chunks earlier. This is the subject of the next lesson.\n\n**Distractors crowded out the answer.** Ten passages retrieved, one correct, nine plausibly similar. The previous lesson showed this measurably reduces accuracy.\n\n**The model ignored the context and used its weights.** It happens, particularly when the retrieved text contradicts something the model believes strongly. Instructing it to answer only from the provided context helps and does not eliminate it.\n\n**The model used the context and the context was wrong.** Your knowledge base contains an outdated policy. The system faithfully reports it, with a citation, and looks entirely trustworthy. Retrieval improves grounding, not truth.\n\n**The question was not a retrieval question.** \"Summarise the themes across all our support tickets this quarter\" cannot be answered by fetching five chunks. Aggregation, counting and comparison need a different tool: usually a query over structured data. Recognising which questions are out of scope is part of the design.\n\n## Building it free\n\nNothing here requires a paid service. `sentence-transformers` for embeddings, `rank_bm25` or SQLite's full-text search for keywords, NumPy for the vector store, `pypdf` for extraction, `llama.cpp` for the model. A working system on a laptop, with no account anywhere, is an afternoon.\n\nStart there even if you intend to buy something later. The version you build yourself teaches you which stage your failures come from, and that knowledge survives every change of vendor.\n\n## Evaluate the stages separately\n\nThe one habit that distinguishes systems that improve from systems that get tinkered with: measure retrieval and generation apart.\n\n- **Retrieval:** for 50 questions with known source chunks, what fraction have the right chunk in the top 5? That is recall, and it caps everything downstream.\n- **Generation:** give the model the *correct* chunk by hand and ask the question. If it still answers badly, the problem is generation. If it answers well, every remaining failure is retrieval.\n\nTen minutes of this will usually redirect a week of work. The evaluation course treats the full method; the point here is that the pipeline has separable stages and that treating it as one black box is why so many retrieval projects stall at \"it mostly works\"."},"check":{"question":"A retrieval system answers 55 per cent of test questions correctly. The team tries three prompt rewrites and gets no improvement. What should they measure?","options":[{"text":"Whether the model is exceeding its effective context length with the retrieved passages.","correct":false,"tempting":"Position effects are real and worth checking, which makes this a well-informed guess at a genuine failure mode."},{"text":"Whether the embedding model matches the language of the documents.","correct":false,"tempting":"Language mismatch does wreck retrieval quality, so it is a sound thing to verify in a multilingual corpus."},{"text":"What fraction of the time the correct chunk appears in the retrieved set at all, since that caps accuracy regardless of the prompt.","correct":true},{"text":"Whether the temperature is high enough to let the model consider alternative answers.","correct":false,"tempting":"Sampling settings do change output, and exploring alternatives sounds like it should help on hard questions."}]},"prev":{"slug":"prompt-caching","title":"Prompt caching, and the ordering it demands","url":"https://addaly.com/learn/how-llms-work/prompt-caching"},"next":{"slug":"chunking-decisions","title":"Chunking: the decision that quietly sets your ceiling","url":"https://addaly.com/learn/how-llms-work/chunking-decisions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Retrieval, as a pipeline that can break in six places', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/retrieval-mechanically (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Retrieval, as a pipeline that can break in six places","courseTitle":"How a Language Model Actually Works","module":{"index":4,"slug":"the-context-window","title":"What the model can see"},"answer":null,"keyIdea":"Retrieval-augmented generation is a search system with a model on the end, and most of its failures are search failures that get blamed on the model.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Split the documents into chunks","detail":"A boundary through a table or an answer caps everything downstream"},{"label":"Embed and store the chunks","detail":"Vectors from one provider’s model are meaningless to another’s"},{"label":"Embed the question","detail":"“The drug is safe” and “the drug is not safe” sit very close together"},{"label":"Retrieve the nearest few","detail":"If the right chunk is in the top five only 60% of the time, nothing above 60% is available"},{"label":"Paste them into the prompt","detail":"Nine plausible distractors around one correct passage measurably lower accuracy"},{"label":"Generate","detail":"The model may answer from its weights instead, or report an outdated policy faithfully and with a citation"}],"title":"Six places retrieval breaks, in order of frequency","caption":"Five of the six failures happen before the model is asked anything, and all six get reported as “the model hallucinated”. Measure the stages apart: hand the model the correct chunk yourself, and whatever still fails is generation."},"alt":"Figure: Six places retrieval breaks, in order of frequency. Steps: 1. Split the documents into chunks (A boundary through a table or an answer caps everything downstream); 2. Embed and store the chunks (Vectors from one provider’s model are meaningless to another’s); 3. Embed the question (“The drug is safe” and “the drug is not safe” sit very close together); 4. Retrieve the nearest few (If the right chunk is in the top five only 60% of the time, nothing above 60% is available); 5. Paste them into the prompt (Nine plausible distractors around one correct passage measurably lower accuracy); 6. Generate (The model may answer from its weights instead, or report an outdated policy faithfully and with a citation). Five of the six failures happen before the model is asked anything, and all six get reported as “the model hallucinated”. Measure the stages apart: hand the model the correct chunk yourself, and whatever still fails is generation."}},"blocks":[{"t":"h2","id":"the-pipeline-stated-plainly","runs":[{"text":"The pipeline, stated plainly"}]},{"t":"p","runs":[{"text":"Retrieval-augmented generation has an aura of sophistication that it does not deserve. It is this:"}]},{"t":"ol","start":1,"items":[[{"text":"Split","b":true},{"text":" your documents into chunks."}],[{"text":"Embed","b":true},{"text":" each chunk and store the vectors."}],[{"text":"Embed","b":true},{"text":" the incoming question."}],[{"text":"Retrieve","b":true},{"text":" the nearest chunks."}],[{"text":"Paste","b":true},{"text":" them into the prompt with an instruction to answer from them."}],[{"text":"Generate.","b":true}]]},{"t":"p","runs":[{"text":"Nothing more. When it works well it is because steps 1 to 4 worked well. When it fails, the failure is usually in steps 1 to 4 and gets described as “the model hallucinated”."}]},{"t":"h2","id":"why-it-exists","runs":[{"text":"Why it exists"}]},{"t":"p","runs":[{"text":"Module one established that facts live in weights, set at training time, and cannot be edited by prompting. Retrieval sidesteps that. Instead of changing what the model knows, you change what it can see. This gives you:"}]},{"t":"ul","items":[[{"text":"Current information","b":true},{"text":" without retraining."}],[{"text":"Private information","b":true},{"text":" the model was never trained on."}],[{"text":"Citations","b":true},{"text":", because you know which chunk each answer came from."}],[{"text":"Revocation","b":true},{"text":", because deleting a document removes it from future answers, which fine-tuning cannot do."}]]},{"t":"p","runs":[{"text":"That last property is underrated and often decisive for anything with a compliance requirement."}]},{"t":"h2","id":"where-it-breaks-in-order-of-frequency","runs":[{"text":"Where it breaks, in order of frequency"}]},{"t":"p","runs":[{"text":"The answer was never retrieved.","b":true},{"text":" The model cannot use what it never received. Measure this separately from everything else: for a set of questions with known source documents, what fraction of the time is the right chunk in the top "},{"text":"k","i":true},{"text":"? If that number is 60 per cent, no prompt engineering will get you above 60 per cent, and you are working on the wrong problem."}]},{"t":"p","runs":[{"text":"The chunk was retrieved but incomplete.","b":true},{"text":" The answer spans a boundary; a table’s header is in one chunk and its rows in another; a pronoun in the chunk refers to a subject named two chunks earlier. This is the subject of the next lesson."}]},{"t":"p","runs":[{"text":"Distractors crowded out the answer.","b":true},{"text":" Ten passages retrieved, one correct, nine plausibly similar. The previous lesson showed this measurably reduces accuracy."}]},{"t":"p","runs":[{"text":"The model ignored the context and used its weights.","b":true},{"text":" It happens, particularly when the retrieved text contradicts something the model believes strongly. Instructing it to answer only from the provided context helps and does not eliminate it."}]},{"t":"p","runs":[{"text":"The model used the context and the context was wrong.","b":true},{"text":" Your knowledge base contains an outdated policy. The system faithfully reports it, with a citation, and looks entirely trustworthy. Retrieval improves grounding, not truth."}]},{"t":"p","runs":[{"text":"The question was not a retrieval question.","b":true},{"text":" “Summarise the themes across all our support tickets this quarter” cannot be answered by fetching five chunks. Aggregation, counting and comparison need a different tool: usually a query over structured data. Recognising which questions are out of scope is part of the design."}]},{"t":"h2","id":"building-it-free","runs":[{"text":"Building it free"}]},{"t":"p","runs":[{"text":"Nothing here requires a paid service. "},{"text":"sentence-transformers","code":true},{"text":" for embeddings, "},{"text":"rank_bm25","code":true},{"text":" or SQLite’s full-text search for keywords, NumPy for the vector store, "},{"text":"pypdf","code":true},{"text":" for extraction, "},{"text":"llama.cpp","code":true},{"text":" for the model. A working system on a laptop, with no account anywhere, is an afternoon."}]},{"t":"p","runs":[{"text":"Start there even if you intend to buy something later. The version you build yourself teaches you which stage your failures come from, and that knowledge survives every change of vendor."}]},{"t":"h2","id":"evaluate-the-stages-separately","runs":[{"text":"Evaluate the stages separately"}]},{"t":"p","runs":[{"text":"The one habit that distinguishes systems that improve from systems that get tinkered with: measure retrieval and generation apart."}]},{"t":"ul","items":[[{"text":"Retrieval:","b":true},{"text":" for 50 questions with known source chunks, what fraction have the right chunk in the top 5? That is recall, and it caps everything downstream."}],[{"text":"Generation:","b":true},{"text":" give the model the "},{"text":"correct","i":true},{"text":" chunk by hand and ask the question. If it still answers badly, the problem is generation. If it answers well, every remaining failure is retrieval."}]]},{"t":"p","runs":[{"text":"Ten minutes of this will usually redirect a week of work. The evaluation course treats the full method; the point here is that the pipeline has separable stages and that treating it as one black box is why so many retrieval projects stall at “it mostly works”."}]}],"check":{"question":[{"text":"A retrieval system answers 55 per cent of test questions correctly. The team tries three prompt rewrites and gets no improvement. What should they measure?"}],"options":[{"text":[{"text":"Whether the model is exceeding its effective context length with the retrieved passages."}],"correct":false,"tempting":[{"text":"Position effects are real and worth checking, which makes this a well-informed guess at a genuine failure mode."}]},{"text":[{"text":"Whether the embedding model matches the language of the documents."}],"correct":false,"tempting":[{"text":"Language mismatch does wreck retrieval quality, so it is a sound thing to verify in a multilingual corpus."}]},{"text":[{"text":"What fraction of the time the correct chunk appears in the retrieved set at all, since that caps accuracy regardless of the prompt."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the temperature is high enough to let the model consider alternative answers."}],"correct":false,"tempting":[{"text":"Sampling settings do change output, and exploring alternatives sounds like it should help on hard questions."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"prompt-caching","title":"Prompt caching, and the ordering it demands","minutes":8},"next":{"slug":"chunking-decisions","title":"Chunking: the decision that quietly sets your ceiling","minutes":8}}}