{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"measuring-retrieval-on-its-own","title":"Measuring retrieval on its own","url":"https://addaly.com/learn/building-with-ai/measuring-retrieval-on-its-own","position":56,"of":80,"minutes":8,"answer":"In a system that retrieves and then generates, a wrong answer has two possible causes, and only a gold set of question-to-chunk pairs with a recall number tells you which.","keyIdea":"In a system that retrieves and then generates, a wrong answer has two possible causes, and only a gold set of question-to-chunk pairs with a recall number tells you which, so build that set of forty questions in an afternoon and report retrieval and generation failures separately."},"body":{"format":"markdown","text":"## Two stages, one wrong answer\n\nA retrieval feature gives a wrong answer. The prompt is blamed, tuned, and the answer is still wrong, because the right passage was never retrieved and no prompt can make the model cite what it did not see. Or the passage was retrieved, and the model ignored it, and a week is spent on the embedding model instead. Every wrong answer from a retrieval-augmented system has two candidate causes, and the only way to tell them apart is to measure the retrieval stage separately from the generation stage.\n\n## The gold set\n\nForty questions, each paired with the ids of the chunks that answer it. Not the answer text: the chunk ids. That is the gold set for retrieval, and it is built by hand: take real questions from the logs and the spec, search the corpus yourself, and record which chunks contain the answer. Some questions have one answering chunk; some have three; a few have none, and those are kept, tagged `unanswerable`, because the correct retrieval result for them is an empty or low-scored list.\n\nIt takes an afternoon. Store it beside the case file from the earlier lesson, with the same provenance field, because chunk ids change when the corpus is re-ingested and you need to know which version of the index the set was built against.\n\n## The two numbers\n\n**Recall at k**: for each question, is at least one of its gold chunks in the top k results? Averaged over the set. At k equal to the number of chunks you send to the model, this is the fraction of questions for which the model was given the answer. If it is 0.7, three questions in ten were unanswerable before generation began, whatever the prompt.\n\n**Mean reciprocal rank**: for each question, one divided by the rank of the first gold chunk, averaged. A system that puts the right chunk first every time scores 1.0; one that puts it fifth scores 0.2 on that question. It tells you whether the right chunk is arriving at the top, where the model attends to it, or at the bottom, where the reranking lesson showed it is often ignored.\n\n```python\ndef recall_at_k(results, gold, k):\n    return sum(any(c in gold[q] for c in results[q][:k]) for q in gold) / len(gold)\n\ndef mrr(results, gold):\n    total = 0\n    for q in gold:\n        ranks = [i for i, c in enumerate(results[q], 1) if c in gold[q]]\n        total += 1 / ranks[0] if ranks else 0\n    return total / len(gold)\n```\n\nThat is the whole measurement. The `ranx` library does it with more metrics and less code, if you prefer.\n\n## The two-by-two that ends the argument\n\nFor every failed case in the end-to-end evaluation, record two facts: was a gold chunk retrieved, and was the answer correct. Four cells.\n\n- Retrieved and correct: fine.\n- Not retrieved and wrong: a retrieval failure. Work on chunking, embeddings, hybrid search, rewriting: the retrieval module.\n- Retrieved and wrong: a generation failure. The model had the answer and did not use it: work on the prompt, the number of chunks, their order, the model.\n- Not retrieved and correct: the model answered from its own knowledge. Convenient today; a hallucination tomorrow when the corpus and the world disagree.\n\nCount the cells. The largest one is where the next week goes. Teams that skip this step tune the wrong stage more often than not, because the prompt is the easiest thing to change and so it is changed first.\n\n## What to run it on\n\nRun the retrieval measurement on every change to the retrieval stage (chunk size, embedding model, index parameters, hybrid weights, reranker, query rewriting) before touching the end-to-end evaluation. It is fast: no generation, forty searches, a second of compute. A change that raises recall at five from 0.72 to 0.85 is worth shipping before you know what it does to the final answer, because the final answer cannot get worse from the model being handed the right passage more often.\n\n## The set goes stale\n\nChunk ids change on re-ingestion; documents are edited; the right answer moves. Re-validate the gold set when the corpus changes materially, and store the index version it was built against. A recall number computed against ids that no longer exist is zero, and it will look like a catastrophic regression rather than a stale file.\n\n## Unanswerable questions\n\nThe `unanswerable` cases are where a threshold is set. For those questions, the top result's score (from the reranker, ideally) should sit below the cutoff that triggers \"I could not find this\", and for the answerable ones it should sit above. Plot the two distributions; the gap, if there is one, is where the cutoff goes. If there is no gap, your system cannot tell when it has nothing, and it will answer confidently from the wrong passage. Knowing that is worth the afternoon on its own.\n\n> Figure: Fifty evaluation cases, sorted by what the retriever did. A two by two grid, rows Gold chunk retrieved and Gold chunk not retrieved, columns Answer correct and Answer wrong: Gold chunk retrieved and Answer correct: 31; Gold chunk retrieved and Answer wrong: 9; Gold chunk not retrieved and Answer correct: 4; Gold chunk not retrieved and Answer wrong: 6. The nine and the six are different bugs with different fixes, and an end-to-end pass rate of 70 per cent cannot tell them apart. The four in the bottom left are the uncomfortable ones: the model answered from training, which is convenient today and a hallucination the day the corpus and the world disagree."},"check":{"question":"An end-to-end evaluation shows 30 per cent of answers wrong. On inspection, in most of those cases the answering chunk was in the model's context. Where should the effort go?","options":[{"text":"Into the retrieval stage: a better embedding model would surface more relevant chunks and reduce the error rate across the whole set.","correct":false,"tempting":"Retrieval already delivered the answer in most failures; improving it would leave those cases unchanged."},{"text":"Into re-chunking the corpus so that answers are more self-contained.","correct":false,"tempting":"Chunking affects what is retrieved, and the right chunk was retrieved; the model failed to use it."},{"text":"Into generation: the model is failing to use a passage it was given, so the prompt, chunk count, ordering or model are the levers.","correct":true},{"text":"Into expanding the gold set, since 40 questions cannot reliably locate a failure.","correct":false,"tempting":"More questions would sharpen the estimate, but the two-by-two already points clearly at one stage."}]},"prev":{"slug":"a-judge-you-have-checked","title":"A model judge you have checked against people","url":"https://addaly.com/learn/building-with-ai/a-judge-you-have-checked"},"next":{"slug":"the-harness-in-ci","title":"The harness in CI, and why temperature zero is not deterministic","url":"https://addaly.com/learn/building-with-ai/the-harness-in-ci"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring retrieval on its own', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/measuring-retrieval-on-its-own (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring retrieval on its own","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"In a system that retrieves and then generates, a wrong answer has two possible causes, and only a gold set of question-to-chunk pairs with a recall number tells you which, so build that set of forty questions in an afternoon and report retrieval and generation failures separately.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Answer correct","Answer wrong"],"rows":["Gold chunk retrieved","Gold chunk not retrieved"],"type":"matrix","cells":[["31","9"],["4","6"]],"title":"Fifty evaluation cases, sorted by what the retriever did","caption":"The nine and the six are different bugs with different fixes, and an end-to-end pass rate of 70 per cent cannot tell them apart. The four in the bottom left are the uncomfortable ones: the model answered from training, which is convenient today and a hallucination the day the corpus and the world disagree.","cellLabels":[["working as designed","generation failure: it had the passage"],["answered from its own memory","retrieval failure: no prompt can fix it"]]},"alt":"Figure: Fifty evaluation cases, sorted by what the retriever did. A two by two grid, rows Gold chunk retrieved and Gold chunk not retrieved, columns Answer correct and Answer wrong: Gold chunk retrieved and Answer correct: 31; Gold chunk retrieved and Answer wrong: 9; Gold chunk not retrieved and Answer correct: 4; Gold chunk not retrieved and Answer wrong: 6. The nine and the six are different bugs with different fixes, and an end-to-end pass rate of 70 per cent cannot tell them apart. The four in the bottom left are the uncomfortable ones: the model answered from training, which is convenient today and a hallucination the day the corpus and the world disagree."}},"blocks":[{"t":"h2","id":"two-stages-one-wrong-answer","runs":[{"text":"Two stages, one wrong answer"}]},{"t":"p","runs":[{"text":"A retrieval feature gives a wrong answer. The prompt is blamed, tuned, and the answer is still wrong, because the right passage was never retrieved and no prompt can make the model cite what it did not see. Or the passage was retrieved, and the model ignored it, and a week is spent on the embedding model instead. Every wrong answer from a retrieval-augmented system has two candidate causes, and the only way to tell them apart is to measure the retrieval stage separately from the generation stage."}]},{"t":"h2","id":"the-gold-set","runs":[{"text":"The gold set"}]},{"t":"p","runs":[{"text":"Forty questions, each paired with the ids of the chunks that answer it. Not the answer text: the chunk ids. That is the gold set for retrieval, and it is built by hand: take real questions from the logs and the spec, search the corpus yourself, and record which chunks contain the answer. Some questions have one answering chunk; some have three; a few have none, and those are kept, tagged "},{"text":"unanswerable","code":true},{"text":", because the correct retrieval result for them is an empty or low-scored list."}]},{"t":"p","runs":[{"text":"It takes an afternoon. Store it beside the case file from the earlier lesson, with the same provenance field, because chunk ids change when the corpus is re-ingested and you need to know which version of the index the set was built against."}]},{"t":"h2","id":"the-two-numbers","runs":[{"text":"The two numbers"}]},{"t":"p","runs":[{"text":"Recall at k","b":true},{"text":": for each question, is at least one of its gold chunks in the top k results? Averaged over the set. At k equal to the number of chunks you send to the model, this is the fraction of questions for which the model was given the answer. If it is 0.7, three questions in ten were unanswerable before generation began, whatever the prompt."}]},{"t":"p","runs":[{"text":"Mean reciprocal rank","b":true},{"text":": for each question, one divided by the rank of the first gold chunk, averaged. A system that puts the right chunk first every time scores 1.0; one that puts it fifth scores 0.2 on that question. It tells you whether the right chunk is arriving at the top, where the model attends to it, or at the bottom, where the reranking lesson showed it is often ignored."}]},{"t":"code","lang":"python","label":"Python","text":"def recall_at_k(results, gold, k):\n    return sum(any(c in gold[q] for c in results[q][:k]) for q in gold) / len(gold)\n\ndef mrr(results, gold):\n    total = 0\n    for q in gold:\n        ranks = [i for i, c in enumerate(results[q], 1) if c in gold[q]]\n        total += 1 / ranks[0] if ranks else 0\n    return total / len(gold)"},{"t":"p","runs":[{"text":"That is the whole measurement. The "},{"text":"ranx","code":true},{"text":" library does it with more metrics and less code, if you prefer."}]},{"t":"h2","id":"the-two-by-two-that-ends-the-argument","runs":[{"text":"The two-by-two that ends the argument"}]},{"t":"p","runs":[{"text":"For every failed case in the end-to-end evaluation, record two facts: was a gold chunk retrieved, and was the answer correct. Four cells."}]},{"t":"ul","items":[[{"text":"Retrieved and correct: fine."}],[{"text":"Not retrieved and wrong: a retrieval failure. Work on chunking, embeddings, hybrid search, rewriting: the retrieval module."}],[{"text":"Retrieved and wrong: a generation failure. The model had the answer and did not use it: work on the prompt, the number of chunks, their order, the model."}],[{"text":"Not retrieved and correct: the model answered from its own knowledge. Convenient today; a hallucination tomorrow when the corpus and the world disagree."}]]},{"t":"p","runs":[{"text":"Count the cells. The largest one is where the next week goes. Teams that skip this step tune the wrong stage more often than not, because the prompt is the easiest thing to change and so it is changed first."}]},{"t":"h2","id":"what-to-run-it-on","runs":[{"text":"What to run it on"}]},{"t":"p","runs":[{"text":"Run the retrieval measurement on every change to the retrieval stage (chunk size, embedding model, index parameters, hybrid weights, reranker, query rewriting) before touching the end-to-end evaluation. It is fast: no generation, forty searches, a second of compute. A change that raises recall at five from 0.72 to 0.85 is worth shipping before you know what it does to the final answer, because the final answer cannot get worse from the model being handed the right passage more often."}]},{"t":"h2","id":"the-set-goes-stale","runs":[{"text":"The set goes stale"}]},{"t":"p","runs":[{"text":"Chunk ids change on re-ingestion; documents are edited; the right answer moves. Re-validate the gold set when the corpus changes materially, and store the index version it was built against. A recall number computed against ids that no longer exist is zero, and it will look like a catastrophic regression rather than a stale file."}]},{"t":"h2","id":"unanswerable-questions","runs":[{"text":"Unanswerable questions"}]},{"t":"p","runs":[{"text":"The "},{"text":"unanswerable","code":true},{"text":" cases are where a threshold is set. For those questions, the top result’s score (from the reranker, ideally) should sit below the cutoff that triggers “I could not find this”, and for the answerable ones it should sit above. Plot the two distributions; the gap, if there is one, is where the cutoff goes. If there is no gap, your system cannot tell when it has nothing, and it will answer confidently from the wrong passage. Knowing that is worth the afternoon on its own."}]}],"check":{"question":[{"text":"An end-to-end evaluation shows 30 per cent of answers wrong. On inspection, in most of those cases the answering chunk was in the model’s context. Where should the effort go?"}],"options":[{"text":[{"text":"Into the retrieval stage: a better embedding model would surface more relevant chunks and reduce the error rate across the whole set."}],"correct":false,"tempting":[{"text":"Retrieval already delivered the answer in most failures; improving it would leave those cases unchanged."}]},{"text":[{"text":"Into re-chunking the corpus so that answers are more self-contained."}],"correct":false,"tempting":[{"text":"Chunking affects what is retrieved, and the right chunk was retrieved; the model failed to use it."}]},{"text":[{"text":"Into generation: the model is failing to use a passage it was given, so the prompt, chunk count, ordering or model are the levers."}],"correct":true,"tempting":null},{"text":[{"text":"Into expanding the gold set, since 40 questions cannot reliably locate a failure."}],"correct":false,"tempting":[{"text":"More questions would sharpen the estimate, but the two-by-two already points clearly at one stage."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"a-judge-you-have-checked","title":"A model judge you have checked against people","minutes":9},"next":{"slug":"the-harness-in-ci","title":"The harness in CI, and why temperature zero is not deterministic","minutes":9}}}