Try the retriever first

The generator is usually innocent

When a retrieval-augmented system answers badly, almost everybody blames the model that wrote the answer. That is the wrong end. Before any of that, measure one number: how often the passage containing the answer was in the context at all.

This is recall at k. Take fifty questions for which you know the correct source passage, run your retriever, and count how often the right passage appears in the top k that you actually pass to the model.

hits = sum(gold_id in [c.id for c in retrieve(q, k=5)] for q, gold_id in pairs)
print(f"recall@5 = {hits}/{len(pairs)} = {hits/len(pairs):.2f}")

If that comes back 0.62, then 38% of your failures were decided before the language model saw anything. No amount of fine-tuning the generator recovers a document that was never in the window. You would be training a model to guess better, which is exactly the behaviour you are trying to stop.

The ceiling nobody computes

Recall@k is a ceiling on end-to-end accuracy. A system with recall@5 of 0.62 cannot exceed 62% correct answers, except by luck or by the model already knowing the fact. Teams routinely spend six weeks fine-tuning a generator to close a gap that sits entirely above them.

Write the ceiling on the whiteboard before you plan anything: our end-to-end accuracy cannot beat 0.62 until retrieval improves.

The cheap retrieval fixes, in order

Most of the gap closes without training anything.

1. Chunking. The single largest lever. Chunks that split a table from its header, or a clause from its definition, make the answer unretrievable and also unusable if retrieved. Try 400-token chunks with 80 tokens of overlap, and chunk on document structure (headings, list items) rather than on a fixed character count.
2. Hybrid search. Dense embeddings miss exact strings: part numbers, error codes, surnames. BM25 catches them and misses paraphrase. Run both and fuse the ranks. Reciprocal rank fusion is six lines of code and typically moves recall@5 by 5 to 15 points on real corpora with identifiers in them.
3. Raise k, then rerank. Retrieve 50, rerank to 5 with a cross-encoder. The reranker sees the query and the passage together instead of comparing two independently-made vectors, which is why it is better and why it is slower.
4. Query rewriting. “What about the second one?” is unretrievable. Rewrite conversational turns into standalone queries before embedding them.

Every one of these is free: rank_bm25 for lexical search, sentence-transformers for embeddings, FAISS or sqlite-vec for the index. A corpus of 100,000 chunks fits in memory on a laptop with 8 GB of RAM.

When you do train something, train the retriever

If retrieval is still short after the cheap fixes, fine-tune the embedding model or the reranker, not the generator. The reasons are arithmetic:

• An embedding model is 30M to 500M parameters. A generator you would actually serve is 7B to 70B. The training job is a hundred times smaller.
• Embedding fine-tuning works on pairs: a query and the passage that answers it. You can mine those from click logs or generate them by asking a model to write a question for each passage. You need hundreds, not tens of thousands.
• A retriever improvement raises the ceiling for every downstream model, including the next one you switch to. A generator fine-tune is welded to one base model and dies with it.

The practice of this is in the module on fine-tuning beyond a chat model. The decision belongs here: retriever before generator, always, unless the retrieval numbers say retrieval is fine.

The limitation

Retrieval cannot fix questions whose answer is not written down anywhere: aggregations (“how many of our contracts expire in Q3”), comparisons across many documents, or reasoning over a whole corpus. Vector search returns the k most similar chunks, and “most similar” has nothing to do with “all the ones you need to count”. Those questions want a database query or a document pipeline, and a team that keeps trying to make embeddings answer them ends up fine-tuning in frustration for a capability neither component has.

So the diagnosis has two steps, not one. First: was the evidence there? If not, fix retrieval. Second: is this question even a retrieval question? If not, neither lever helps, and you need a different system shape.