Retrieval, as a pipeline that can break in six places

The pipeline, stated plainly

Retrieval-augmented generation has an aura of sophistication that it does not deserve. It is this:

1. Split your documents into chunks.
2. Embed each chunk and store the vectors.
3. Embed the incoming question.
4. Retrieve the nearest chunks.
5. Paste them into the prompt with an instruction to answer from them.
6. Generate.

Nothing more. When it works well it is because steps 1 to 4 worked well. When it fails, the failure is usually in steps 1 to 4 and gets described as “the model hallucinated”.

Why it exists

Module one established that facts live in weights, set at training time, and cannot be edited by prompting. Retrieval sidesteps that. Instead of changing what the model knows, you change what it can see. This gives you:

• Current information without retraining.
• Private information the model was never trained on.
• Citations, because you know which chunk each answer came from.
• Revocation, because deleting a document removes it from future answers, which fine-tuning cannot do.

That last property is underrated and often decisive for anything with a compliance requirement.

Where it breaks, in order of frequency

The answer was never retrieved. The model cannot use what it never received. Measure this separately from everything else: for a set of questions with known source documents, what fraction of the time is the right chunk in the top k? If that number is 60 per cent, no prompt engineering will get you above 60 per cent, and you are working on the wrong problem.

The chunk was retrieved but incomplete. The answer spans a boundary; a table’s header is in one chunk and its rows in another; a pronoun in the chunk refers to a subject named two chunks earlier. This is the subject of the next lesson.

Distractors crowded out the answer. Ten passages retrieved, one correct, nine plausibly similar. The previous lesson showed this measurably reduces accuracy.

The model ignored the context and used its weights. It happens, particularly when the retrieved text contradicts something the model believes strongly. Instructing it to answer only from the provided context helps and does not eliminate it.

The model used the context and the context was wrong. Your knowledge base contains an outdated policy. The system faithfully reports it, with a citation, and looks entirely trustworthy. Retrieval improves grounding, not truth.

The question was not a retrieval question. “Summarise the themes across all our support tickets this quarter” cannot be answered by fetching five chunks. Aggregation, counting and comparison need a different tool: usually a query over structured data. Recognising which questions are out of scope is part of the design.

Building it free

Nothing here requires a paid service. sentence-transformers for embeddings, rank_bm25 or SQLite’s full-text search for keywords, NumPy for the vector store, pypdf for extraction, llama.cpp for the model. A working system on a laptop, with no account anywhere, is an afternoon.

Start there even if you intend to buy something later. The version you build yourself teaches you which stage your failures come from, and that knowledge survives every change of vendor.

Evaluate the stages separately

The one habit that distinguishes systems that improve from systems that get tinkered with: measure retrieval and generation apart.

• Retrieval: for 50 questions with known source chunks, what fraction have the right chunk in the top 5? That is recall, and it caps everything downstream.
• Generation: give the model the correct chunk by hand and ask the question. If it still answers badly, the problem is generation. If it answers well, every remaining failure is retrieval.

Ten minutes of this will usually redirect a week of work. The evaluation course treats the full method; the point here is that the pipeline has separable stages and that treating it as one black box is why so many retrieval projects stall at “it mostly works”.