{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"measuring-retrieval-on-its-own","title":"Measuring retrieval without the model","url":"https://addaly.com/learn/context-engineering/measuring-retrieval-on-its-own","position":30,"of":87,"minutes":9,"answer":"Score retrieval separately with recall@k on a gold set (it costs no model calls, it caps everything downstream, and it splits failures into missing evidence and unused evidence) but discount recall measured on model-generated questions.","keyIdea":"Score retrieval separately with recall@k on a gold set (it costs no model calls, it caps everything downstream, and it splits failures into missing evidence and unused evidence) but discount recall measured on model-generated questions, which reuse the chunk's own words."},"body":{"format":"markdown","text":"## The number that lets you stop guessing\n\nEnd-to-end accuracy tells you the system is wrong. It does not tell you which half. Retrieval metrics, measured on their own, split every failure into \"the evidence was not there\" and \"the evidence was there and was not used\", and that split determines which module of this course you should be working in.\n\nIt is also cheap. Retrieval scoring needs no model call, so a full run over 500 queries takes seconds and costs nothing. You can put it in CI.\n\n> Figure: Which half of the system to work on. A two by two grid, rows Answer correct and Answer wrong, columns Evidence retrieved and Evidence missing: Answer correct and Evidence retrieved: Working; Answer correct and Evidence missing: Lucky; Answer wrong and Evidence retrieved: Assembly problem; Answer wrong and Evidence missing: Retrieval problem. Retrieval scoring needs no model call, so a run over five hundred queries takes seconds and costs nothing, which is why it belongs in continuous integration. Ninety per cent recall with sixty per cent answer accuracy is an assembly problem, and no amount of retriever tuning will move it.\n\n## The three metrics, and when each is the right one\n\n**Recall@k**: in what fraction of queries is at least one relevant passage inside the top k? This is the one that matters most, because it is the ceiling on everything downstream. If recall@10 is 0.72, no reranker, no prompt and no model gets you above 72% on questions needing retrieval.\n\n**MRR**: the mean of one over the rank of the first relevant passage. Sensitive to whether the right answer is first or fourth. Worth watching when you show a top result directly, less important when you pass ten passages to a model that reads all of them.\n\n**nDCG@k**: accounts for graded relevance and position. Use it when passages are not simply relevant or irrelevant, for example a search interface where the ordering itself is the product.\n\nFor most retrieval-augmented systems, track recall@k at the k you actually use, plus recall at the depth you rerank from. Those two numbers between them tell you whether to work on the first stage or the second.\n\n## Building a gold set without a labelling budget\n\nYou need queries paired with the passage that answers them. Four sources, best first:\n\n**1. Resolved support tickets or answered questions.** The question is real and somebody already found the document. This is the highest-quality source and most organisations are sitting on thousands of them without realising they are a test set.\n\n**2. Search logs with a click.** The query is real; the click is a noisy relevance label. Noisy is fine in aggregate for tracking movement over time.\n\n**3. Questions written by people who know the domain.** Fifty from a support lead is an afternoon's work and better than any synthetic set.\n\n**4. Model-generated questions from your own chunks.** Take a chunk, ask a model to write a question it answers, and you have a pair. Fast, free, scales to thousands.\n\nOption four comes with a caveat you must apply. A question generated *from* a chunk reuses the chunk's own vocabulary, so it is unnaturally easy for keyword retrieval and easy for dense retrieval too. Recall measured on a generated set is systematically higher than recall on real user queries, commonly by a wide margin.\n\nTwo mitigations. Instruct the generator to write as a user who has not read the document would, using everyday words. And always keep a smaller real set alongside, so you can see how far apart the two numbers are. If your synthetic recall is 0.94 and your real recall is 0.68, quote the second.\n\n## Beware the passage that is relevant and not gold\n\nA gold set with one correct passage per query will mark a correct retrieval as a miss whenever the corpus has more than one passage answering the question, which, after the duplication lesson, you know is common.\n\nThe practical fix is to allow multiple gold ids per query, and to add them as you find them. When investigating a supposed miss, if the retrieved passage does answer the question, add it to the gold list rather than accepting the false negative. The set improves every time you look at it, and this is one of the few kinds of maintenance that compounds.\n\n## The harness\n\n```python\ndef recall_at(retriever, gold, ks=(1, 5, 10, 50)):\n    hits = {k: 0 for k in ks}\n    for q, gold_ids in gold:\n        ranked = [d.id for d in retriever(q, k=max(ks))]\n        for k in ks:\n            if set(ranked[:k]) & set(gold_ids):\n                hits[k] += 1\n    return {f\"recall@{k}\": hits[k] / len(gold) for k in ks}\n```\n\nRun it on every change to chunking, embeddings, the query rewriter, the fusion, the filters. Each of those is a change to retrieval and each can be checked in seconds without touching a language model.\n\n## Slice it, always\n\nOne average hides everything interesting. Report recall broken down by query type (identifier lookups, conceptual questions, comparisons) and by language.\n\nThat is where hybrid search shows its real value, where a multilingual embedding model's weakness in one script becomes visible, and where you find that the system works well for the questions you designed for and poorly for the ones people ask. An overall recall of 0.85 made of 0.95 on head queries and 0.55 on the tail is a different system from a uniform 0.85, and only one of the two is ready to ship."},"check":{"question":"A team reports recall@10 of 0.93 on a set of questions generated from their own chunks, yet users still frequently get \"I don't have that information\". What is the most likely explanation?","options":[{"text":"Generated questions reuse the source chunk's vocabulary, so they overstate recall relative to how real users phrase things.","correct":true},{"text":"Recall@10 is the wrong metric and nDCG would reveal the problem.","correct":false,"tempting":"Changes the metric rather than the set; a graded metric on the same easy questions would also look good."},{"text":"The abstention threshold is set too high, so good retrievals are being discarded before generation.","correct":false,"tempting":"Worth checking and a real possibility, but it would not reconcile a synthetic recall figure with real-user behaviour."},{"text":"The gold set allows only one correct passage per query, so recall is understated rather than overstated.","correct":false,"tempting":"Identifies a genuine gold-set flaw, but it biases the number downwards, which is the opposite of the discrepancy described."}]},"prev":{"slug":"citations-that-survive-assembly","title":"Citations you can actually verify","url":"https://addaly.com/learn/context-engineering/citations-that-survive-assembly"},"next":{"slug":"lost-in-the-middle-the-experiment","title":"The experiment that named the problem","url":"https://addaly.com/learn/context-engineering/lost-in-the-middle-the-experiment"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring retrieval without the model', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/measuring-retrieval-on-its-own (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring retrieval without the model","courseTitle":"Context Engineering","module":{"index":3,"slug":"retrieval-as-a-pipeline","title":"Retrieval as a context pipeline"},"answer":null,"keyIdea":"Score retrieval separately with recall@k on a gold set (it costs no model calls, it caps everything downstream, and it splits failures into missing evidence and unused evidence) but discount recall measured on model-generated questions, which reuse the chunk’s own words.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Evidence retrieved","Evidence missing"],"rows":["Answer correct","Answer wrong"],"type":"matrix","cells":[["Working","Lucky"],["Assembly problem","Retrieval problem"]],"title":"Which half of the system to work on","caption":"Retrieval scoring needs no model call, so a run over five hundred queries takes seconds and costs nothing, which is why it belongs in continuous integration. Ninety per cent recall with sixty per cent answer accuracy is an assembly problem, and no amount of retriever tuning will move it.","cellLabels":[["the passage was there and used","answered from memory, do not count it"],["position, distractors, instruction","chunking, hybrid search, rewriting"]]},"alt":"Figure: Which half of the system to work on. A two by two grid, rows Answer correct and Answer wrong, columns Evidence retrieved and Evidence missing: Answer correct and Evidence retrieved: Working; Answer correct and Evidence missing: Lucky; Answer wrong and Evidence retrieved: Assembly problem; Answer wrong and Evidence missing: Retrieval problem. Retrieval scoring needs no model call, so a run over five hundred queries takes seconds and costs nothing, which is why it belongs in continuous integration. Ninety per cent recall with sixty per cent answer accuracy is an assembly problem, and no amount of retriever tuning will move it."}},"blocks":[{"t":"h2","id":"the-number-that-lets-you-stop-guessing","runs":[{"text":"The number that lets you stop guessing"}]},{"t":"p","runs":[{"text":"End-to-end accuracy tells you the system is wrong. It does not tell you which half. Retrieval metrics, measured on their own, split every failure into “the evidence was not there” and “the evidence was there and was not used”, and that split determines which module of this course you should be working in."}]},{"t":"p","runs":[{"text":"It is also cheap. Retrieval scoring needs no model call, so a full run over 500 queries takes seconds and costs nothing. You can put it in CI."}]},{"t":"h2","id":"the-three-metrics-and-when-each-is-the-right-one","runs":[{"text":"The three metrics, and when each is the right one"}]},{"t":"p","runs":[{"text":"Recall@k","b":true},{"text":": in what fraction of queries is at least one relevant passage inside the top k? This is the one that matters most, because it is the ceiling on everything downstream. If recall@10 is 0.72, no reranker, no prompt and no model gets you above 72% on questions needing retrieval."}]},{"t":"p","runs":[{"text":"MRR","b":true},{"text":": the mean of one over the rank of the first relevant passage. Sensitive to whether the right answer is first or fourth. Worth watching when you show a top result directly, less important when you pass ten passages to a model that reads all of them."}]},{"t":"p","runs":[{"text":"nDCG@k","b":true},{"text":": accounts for graded relevance and position. Use it when passages are not simply relevant or irrelevant, for example a search interface where the ordering itself is the product."}]},{"t":"p","runs":[{"text":"For most retrieval-augmented systems, track recall@k at the k you actually use, plus recall at the depth you rerank from. Those two numbers between them tell you whether to work on the first stage or the second."}]},{"t":"h2","id":"building-a-gold-set-without-a-labelling-budget","runs":[{"text":"Building a gold set without a labelling budget"}]},{"t":"p","runs":[{"text":"You need queries paired with the passage that answers them. Four sources, best first:"}]},{"t":"p","runs":[{"text":"1. Resolved support tickets or answered questions.","b":true},{"text":" The question is real and somebody already found the document. This is the highest-quality source and most organisations are sitting on thousands of them without realising they are a test set."}]},{"t":"p","runs":[{"text":"2. Search logs with a click.","b":true},{"text":" The query is real; the click is a noisy relevance label. Noisy is fine in aggregate for tracking movement over time."}]},{"t":"p","runs":[{"text":"3. Questions written by people who know the domain.","b":true},{"text":" Fifty from a support lead is an afternoon’s work and better than any synthetic set."}]},{"t":"p","runs":[{"text":"4. Model-generated questions from your own chunks.","b":true},{"text":" Take a chunk, ask a model to write a question it answers, and you have a pair. Fast, free, scales to thousands."}]},{"t":"p","runs":[{"text":"Option four comes with a caveat you must apply. A question generated "},{"text":"from","i":true},{"text":" a chunk reuses the chunk’s own vocabulary, so it is unnaturally easy for keyword retrieval and easy for dense retrieval too. Recall measured on a generated set is systematically higher than recall on real user queries, commonly by a wide margin."}]},{"t":"p","runs":[{"text":"Two mitigations. Instruct the generator to write as a user who has not read the document would, using everyday words. And always keep a smaller real set alongside, so you can see how far apart the two numbers are. If your synthetic recall is 0.94 and your real recall is 0.68, quote the second."}]},{"t":"h2","id":"beware-the-passage-that-is-relevant-and-not-gold","runs":[{"text":"Beware the passage that is relevant and not gold"}]},{"t":"p","runs":[{"text":"A gold set with one correct passage per query will mark a correct retrieval as a miss whenever the corpus has more than one passage answering the question, which, after the duplication lesson, you know is common."}]},{"t":"p","runs":[{"text":"The practical fix is to allow multiple gold ids per query, and to add them as you find them. When investigating a supposed miss, if the retrieved passage does answer the question, add it to the gold list rather than accepting the false negative. The set improves every time you look at it, and this is one of the few kinds of maintenance that compounds."}]},{"t":"h2","id":"the-harness","runs":[{"text":"The harness"}]},{"t":"code","lang":"python","label":"Python","text":"def recall_at(retriever, gold, ks=(1, 5, 10, 50)):\n    hits = {k: 0 for k in ks}\n    for q, gold_ids in gold:\n        ranked = [d.id for d in retriever(q, k=max(ks))]\n        for k in ks:\n            if set(ranked[:k]) & set(gold_ids):\n                hits[k] += 1\n    return {f\"recall@{k}\": hits[k] / len(gold) for k in ks}"},{"t":"p","runs":[{"text":"Run it on every change to chunking, embeddings, the query rewriter, the fusion, the filters. Each of those is a change to retrieval and each can be checked in seconds without touching a language model."}]},{"t":"h2","id":"slice-it-always","runs":[{"text":"Slice it, always"}]},{"t":"p","runs":[{"text":"One average hides everything interesting. Report recall broken down by query type (identifier lookups, conceptual questions, comparisons) and by language."}]},{"t":"p","runs":[{"text":"That is where hybrid search shows its real value, where a multilingual embedding model’s weakness in one script becomes visible, and where you find that the system works well for the questions you designed for and poorly for the ones people ask. An overall recall of 0.85 made of 0.95 on head queries and 0.55 on the tail is a different system from a uniform 0.85, and only one of the two is ready to ship."}]}],"check":{"question":[{"text":"A team reports recall@10 of 0.93 on a set of questions generated from their own chunks, yet users still frequently get “I don’t have that information”. What is the most likely explanation?"}],"options":[{"text":[{"text":"Generated questions reuse the source chunk’s vocabulary, so they overstate recall relative to how real users phrase things."}],"correct":true,"tempting":null},{"text":[{"text":"Recall@10 is the wrong metric and nDCG would reveal the problem."}],"correct":false,"tempting":[{"text":"Changes the metric rather than the set; a graded metric on the same easy questions would also look good."}]},{"text":[{"text":"The abstention threshold is set too high, so good retrievals are being discarded before generation."}],"correct":false,"tempting":[{"text":"Worth checking and a real possibility, but it would not reconcile a synthetic recall figure with real-user behaviour."}]},{"text":[{"text":"The gold set allows only one correct passage per query, so recall is understated rather than overstated."}],"correct":false,"tempting":[{"text":"Identifies a genuine gold-set flaw, but it biases the number downwards, which is the opposite of the discrepancy described."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"citations-that-survive-assembly","title":"Citations you can actually verify","minutes":8},"next":{"slug":"lost-in-the-middle-the-experiment","title":"The experiment that named the problem","minutes":9}}}