{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"try-the-retriever-first","title":"Try the retriever first","url":"https://addaly.com/learn/fine-tuning/try-the-retriever-first","position":4,"of":84,"minutes":9,"answer":"Recall@k is a hard ceiling on a RAG system's accuracy, so measure it before blaming the generator, and when something must be trained, the retriever is a hundred times smaller and outlives the base model.","keyIdea":"Recall@k is a hard ceiling on a RAG system's accuracy, so measure it before blaming the generator, and when something must be trained, the retriever is a hundred times smaller and outlives the base model."},"body":{"format":"markdown","text":"## The generator is usually innocent\n\nWhen a retrieval-augmented system answers badly, almost everybody blames the model that wrote the answer. That is the wrong end. Before any of that, measure one number: how often the passage containing the answer was in the context at all.\n\nThis is recall at k. Take fifty questions for which you know the correct source passage, run your retriever, and count how often the right passage appears in the top k that you actually pass to the model.\n\n```python\nhits = sum(gold_id in [c.id for c in retrieve(q, k=5)] for q, gold_id in pairs)\nprint(f\"recall@5 = {hits}/{len(pairs)} = {hits/len(pairs):.2f}\")\n```\n\nIf that comes back 0.62, then 38% of your failures were decided before the language model saw anything. No amount of fine-tuning the generator recovers a document that was never in the window. You would be training a model to guess better, which is exactly the behaviour you are trying to stop.\n\n## The ceiling nobody computes\n\nRecall@k is a ceiling on end-to-end accuracy. A system with recall@5 of 0.62 cannot exceed 62% correct answers, except by luck or by the model already knowing the fact. Teams routinely spend six weeks fine-tuning a generator to close a gap that sits entirely above them.\n\nWrite the ceiling on the whiteboard before you plan anything: *our end-to-end accuracy cannot beat 0.62 until retrieval improves.*\n\n## The cheap retrieval fixes, in order\n\nMost of the gap closes without training anything.\n\n1. **Chunking.** The single largest lever. Chunks that split a table from its header, or a clause from its definition, make the answer unretrievable and also unusable if retrieved. Try 400-token chunks with 80 tokens of overlap, and chunk on document structure (headings, list items) rather than on a fixed character count.\n2. **Hybrid search.** Dense embeddings miss exact strings: part numbers, error codes, surnames. BM25 catches them and misses paraphrase. Run both and fuse the ranks. Reciprocal rank fusion is six lines of code and typically moves recall@5 by 5 to 15 points on real corpora with identifiers in them.\n3. **Raise k, then rerank.** Retrieve 50, rerank to 5 with a cross-encoder. The reranker sees the query and the passage together instead of comparing two independently-made vectors, which is why it is better and why it is slower.\n4. **Query rewriting.** \"What about the second one?\" is unretrievable. Rewrite conversational turns into standalone queries before embedding them.\n\nEvery one of these is free: `rank_bm25` for lexical search, `sentence-transformers` for embeddings, FAISS or `sqlite-vec` for the index. A corpus of 100,000 chunks fits in memory on a laptop with 8 GB of RAM.\n\n## When you do train something, train the retriever\n\nIf retrieval is still short after the cheap fixes, fine-tune the embedding model or the reranker, not the generator. The reasons are arithmetic:\n\n- An embedding model is 30M to 500M parameters. A generator you would actually serve is 7B to 70B. The training job is a hundred times smaller.\n- Embedding fine-tuning works on pairs: a query and the passage that answers it. You can mine those from click logs or generate them by asking a model to write a question for each passage. You need hundreds, not tens of thousands.\n- A retriever improvement raises the ceiling for *every* downstream model, including the next one you switch to. A generator fine-tune is welded to one base model and dies with it.\n\nThe practice of this is in the module on fine-tuning beyond a chat model. The decision belongs here: retriever before generator, always, unless the retrieval numbers say retrieval is fine.\n\n## The limitation\n\nRetrieval cannot fix questions whose answer is not written down anywhere: aggregations (\"how many of our contracts expire in Q3\"), comparisons across many documents, or reasoning over a whole corpus. Vector search returns the *k* most similar chunks, and \"most similar\" has nothing to do with \"all the ones you need to count\". Those questions want a database query or a document pipeline, and a team that keeps trying to make embeddings answer them ends up fine-tuning in frustration for a capability neither component has.\n\nSo the diagnosis has two steps, not one. First: was the evidence there? If not, fix retrieval. Second: is this question even a retrieval question? If not, neither lever helps, and you need a different system shape."},"check":{"question":"A team measures recall@5 at 0.62 on fifty questions with known gold passages, then fine-tunes their 8B generator on 4,000 examples and finds end-to-end accuracy stuck near 60%. What is the mechanism behind the ceiling?","options":[{"text":"The adapter rank was too low to store the corpus, so a rank of 128 or full fine-tuning would push past 62%.","correct":false,"tempting":"Assumes weights are a storage medium for documents; raising rank increases capacity for behaviour, not a searchable copy of the corpus."},{"text":"In 38% of cases the answer passage was never in the context, so the generator had nothing correct to condition on.","correct":true},{"text":"Four thousand examples is too few; the relationship between dataset size and accuracy means 40,000 would break the ceiling.","correct":false,"tempting":"Treats a missing-evidence problem as a sample-size problem; more examples teach shape, not the contents of documents that were absent."},{"text":"Training on retrieved contexts causes the model to over-trust retrieval, which suppresses its own parametric knowledge and caps accuracy.","correct":false,"tempting":"Names a real effect (grounding suppresses recall of parametric facts) but it is not what pins accuracy at exactly the recall figure."}]},"prev":{"slug":"diagnosing-the-failure","title":"Diagnosing the failure before choosing the fix","url":"https://addaly.com/learn/fine-tuning/diagnosing-the-failure"},"next":{"slug":"classifiers-and-encoders","title":"When the answer is one of six labels","url":"https://addaly.com/learn/fine-tuning/classifiers-and-encoders"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Try the retriever first', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/try-the-retriever-first (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Try the retriever first","courseTitle":"Fine-Tuning, and When Not To","module":{"index":1,"slug":"deciding-whether-to-fine-tune","title":"Deciding whether to fine-tune at all"},"answer":null,"keyIdea":"Recall@k is a hard ceiling on a RAG system’s accuracy, so measure it before blaming the generator, and when something must be trained, the retriever is a hundred times smaller and outlives the base model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-generator-is-usually-innocent","runs":[{"text":"The generator is usually innocent"}]},{"t":"p","runs":[{"text":"When a retrieval-augmented system answers badly, almost everybody blames the model that wrote the answer. That is the wrong end. Before any of that, measure one number: how often the passage containing the answer was in the context at all."}]},{"t":"p","runs":[{"text":"This is recall at k. Take fifty questions for which you know the correct source passage, run your retriever, and count how often the right passage appears in the top k that you actually pass to the model."}]},{"t":"code","lang":"python","label":"Python","text":"hits = sum(gold_id in [c.id for c in retrieve(q, k=5)] for q, gold_id in pairs)\nprint(f\"recall@5 = {hits}/{len(pairs)} = {hits/len(pairs):.2f}\")"},{"t":"p","runs":[{"text":"If that comes back 0.62, then 38% of your failures were decided before the language model saw anything. No amount of fine-tuning the generator recovers a document that was never in the window. You would be training a model to guess better, which is exactly the behaviour you are trying to stop."}]},{"t":"h2","id":"the-ceiling-nobody-computes","runs":[{"text":"The ceiling nobody computes"}]},{"t":"p","runs":[{"text":"Recall@k is a ceiling on end-to-end accuracy. A system with recall@5 of 0.62 cannot exceed 62% correct answers, except by luck or by the model already knowing the fact. Teams routinely spend six weeks fine-tuning a generator to close a gap that sits entirely above them."}]},{"t":"p","runs":[{"text":"Write the ceiling on the whiteboard before you plan anything: "},{"text":"our end-to-end accuracy cannot beat 0.62 until retrieval improves.","i":true}]},{"t":"h2","id":"the-cheap-retrieval-fixes-in-order","runs":[{"text":"The cheap retrieval fixes, in order"}]},{"t":"p","runs":[{"text":"Most of the gap closes without training anything."}]},{"t":"ol","start":1,"items":[[{"text":"Chunking.","b":true},{"text":" The single largest lever. Chunks that split a table from its header, or a clause from its definition, make the answer unretrievable and also unusable if retrieved. Try 400-token chunks with 80 tokens of overlap, and chunk on document structure (headings, list items) rather than on a fixed character count."}],[{"text":"Hybrid search.","b":true},{"text":" Dense embeddings miss exact strings: part numbers, error codes, surnames. BM25 catches them and misses paraphrase. Run both and fuse the ranks. Reciprocal rank fusion is six lines of code and typically moves recall@5 by 5 to 15 points on real corpora with identifiers in them."}],[{"text":"Raise k, then rerank.","b":true},{"text":" Retrieve 50, rerank to 5 with a cross-encoder. The reranker sees the query and the passage together instead of comparing two independently-made vectors, which is why it is better and why it is slower."}],[{"text":"Query rewriting.","b":true},{"text":" “What about the second one?” is unretrievable. Rewrite conversational turns into standalone queries before embedding them."}]]},{"t":"p","runs":[{"text":"Every one of these is free: "},{"text":"rank_bm25","code":true},{"text":" for lexical search, "},{"text":"sentence-transformers","code":true},{"text":" for embeddings, FAISS or "},{"text":"sqlite-vec","code":true},{"text":" for the index. A corpus of 100,000 chunks fits in memory on a laptop with 8 GB of RAM."}]},{"t":"h2","id":"when-you-do-train-something-train-the-retriever","runs":[{"text":"When you do train something, train the retriever"}]},{"t":"p","runs":[{"text":"If retrieval is still short after the cheap fixes, fine-tune the embedding model or the reranker, not the generator. The reasons are arithmetic:"}]},{"t":"ul","items":[[{"text":"An embedding model is 30M to 500M parameters. A generator you would actually serve is 7B to 70B. The training job is a hundred times smaller."}],[{"text":"Embedding fine-tuning works on pairs: a query and the passage that answers it. You can mine those from click logs or generate them by asking a model to write a question for each passage. You need hundreds, not tens of thousands."}],[{"text":"A retriever improvement raises the ceiling for "},{"text":"every","i":true},{"text":" downstream model, including the next one you switch to. A generator fine-tune is welded to one base model and dies with it."}]]},{"t":"p","runs":[{"text":"The practice of this is in the module on fine-tuning beyond a chat model. The decision belongs here: retriever before generator, always, unless the retrieval numbers say retrieval is fine."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"Retrieval cannot fix questions whose answer is not written down anywhere: aggregations (“how many of our contracts expire in Q3”), comparisons across many documents, or reasoning over a whole corpus. Vector search returns the "},{"text":"k","i":true},{"text":" most similar chunks, and “most similar” has nothing to do with “all the ones you need to count”. Those questions want a database query or a document pipeline, and a team that keeps trying to make embeddings answer them ends up fine-tuning in frustration for a capability neither component has."}]},{"t":"p","runs":[{"text":"So the diagnosis has two steps, not one. First: was the evidence there? If not, fix retrieval. Second: is this question even a retrieval question? If not, neither lever helps, and you need a different system shape."}]}],"check":{"question":[{"text":"A team measures recall@5 at 0.62 on fifty questions with known gold passages, then fine-tunes their 8B generator on 4,000 examples and finds end-to-end accuracy stuck near 60%. What is the mechanism behind the ceiling?"}],"options":[{"text":[{"text":"The adapter rank was too low to store the corpus, so a rank of 128 or full fine-tuning would push past 62%."}],"correct":false,"tempting":[{"text":"Assumes weights are a storage medium for documents; raising rank increases capacity for behaviour, not a searchable copy of the corpus."}]},{"text":[{"text":"In 38% of cases the answer passage was never in the context, so the generator had nothing correct to condition on."}],"correct":true,"tempting":null},{"text":[{"text":"Four thousand examples is too few; the relationship between dataset size and accuracy means 40,000 would break the ceiling."}],"correct":false,"tempting":[{"text":"Treats a missing-evidence problem as a sample-size problem; more examples teach shape, not the contents of documents that were absent."}]},{"text":[{"text":"Training on retrieved contexts causes the model to over-trust retrieval, which suppresses its own parametric knowledge and caps accuracy."}],"correct":false,"tempting":[{"text":"Names a real effect (grounding suppresses recall of parametric facts) but it is not what pins accuracy at exactly the recall figure."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"diagnosing-the-failure","title":"Diagnosing the failure before choosing the fix","minutes":10},"next":{"slug":"classifiers-and-encoders","title":"When the answer is one of six labels","minutes":9}}}