{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"retrieval-metrics","title":"Scoring a ranked list","url":"https://addaly.com/learn/evaluating-ai/retrieval-metrics","position":18,"of":76,"minutes":9,"answer":"For a retrieval-augmented system, measure recall at the number of passages you actually pass to the model, because the generator cannot recover an answer that was never retrieved.","keyIdea":"For a retrieval-augmented system, measure recall at the number of passages you actually pass to the model, because the generator cannot recover an answer that was never retrieved."},"body":{"format":"markdown","text":"## When the output is an ordered list\n\nSearch, recommendation and the retrieval half of any RAG system all produce the same shape of output: a ranked list, of which the user or the model downstream sees the top few. Precision and recall need adapting, because position matters: the right document at rank 1 and the right document at rank 9 are not the same result.\n\nSet up the evaluation the same way as everything else in this course. Fifty to a hundred real questions, and for each one, the identifiers of the documents that contain the answer. Building that mapping is the work; once you have it, every metric below is arithmetic.\n\nA note on collecting it: do not write the questions by reading your documents. You will write questions shaped like the text, and the system will look better than it is. Take real questions (from search logs, support tickets, the sales inbox) and then find which document answers each one.\n\n## The four metrics, worked\n\nA query with two relevant documents, `D3` and `D7`. The system returns:\n\n```\nrank 1: D9   (not relevant)\nrank 2: D3   (relevant)\nrank 3: D5   (not relevant)\nrank 4: D2   (not relevant)\nrank 5: D7   (relevant)\n```\n\n**Recall@k**: what fraction of the relevant documents appear in the top k?\n\n```\nrecall@3 = 1/2 = 0.50\nrecall@5 = 2/2 = 1.00\n```\n\n**Precision@k**: what fraction of the top k are relevant?\n\n```\nprecision@3 = 1/3 = 0.33\nprecision@5 = 2/5 = 0.40\n```\n\n**MRR**: reciprocal rank of the *first* relevant result, averaged across queries. Here the first relevant hit is at rank 2, so `1/2 = 0.5`. Use MRR when the user needs one good answer and will stop reading once they find it.\n\n**nDCG@k**: discounted cumulative gain, normalised. It gives credit that falls off with position (`1/log2(rank+1)`) and, unlike the others, handles graded relevance: a document can be perfect, partly useful or useless rather than only relevant or not. Use it when relevance is a matter of degree, and be aware that it is the metric people quote most and inspect least.\n\n## For RAG, recall@k is the one that matters\n\nHere is the practical instruction. If you are retrieving passages to put into a model's context, measure **recall@k where k is the number of passages you actually pass in**. Nothing else comes close in importance.\n\nThe reason is a hard constraint. If the answer is not in the retrieved context, no prompt, no model upgrade and no clever instruction can produce a correct grounded answer. The generation step cannot recover information that was never handed to it. It can only decline, or invent.\n\nTeams routinely spend three weeks rewriting prompts to fix a system whose retrieval recall@5 is 0.55. Measure it first. It takes an afternoon, and about half the time it ends the investigation on the spot.\n\nOrdering *within* the retrieved set matters less than people expect for RAG, because the model reads all of it, though not evenly: content in the middle of a long context is used less reliably than content at either end. Keep k modest and put the strongest passage first.\n\n## Where retrieval fails\n\nWhen recall@5 is poor, the cause is usually one of four things, and they are distinguishable by reading twenty failed queries:\n\n- **Chunking.** The answer spans a boundary, so no single chunk contains it. Overlapping windows, or chunking on document structure rather than a fixed character count, fixes more retrieval problems than any embedding-model change.\n- **Vocabulary mismatch.** The user says \"can't log in\", the document says \"authentication failure\". Dense embeddings help here and are exactly why they exist, but they are weak in the opposite case.\n- **Exact tokens.** Part numbers, error codes, names, dates. Embeddings blur these; keyword search does not miss them. Hybrid retrieval (BM25 and dense vectors, results merged with reciprocal rank fusion) beats either alone on most real corpora, and BM25 is free in `rank_bm25`, SQLite FTS5 or Postgres full-text search.\n- **The document does not exist.** Roughly one failure in five, in my experience of looking, is a question your corpus cannot answer. That is a content problem, and no retrieval work will fix it. It also means your system needs a correct way to say so, which is the subject of the next lesson.\n\n## Tools\n\n`ranx` and `pytrec_eval` compute all four metrics from a run file and are free and open source. BEIR is a standard suite of retrieval benchmarks, useful for choosing an embedding model to start from, useless as a substitute for measuring on your own corpus. Honestly, twenty lines of Python over a dictionary of `{query_id: [relevant_doc_ids]}` will do everything in this lesson, and writing it yourself means you know what the numbers mean."},"check":{"question":"A RAG assistant answers 58% of questions correctly. The team is comparing three prompt rewrites to improve it. What should they measure first instead?","options":[{"text":"Recall@k at the number of passages fed to the model, since a correct answer is impossible when the supporting passage was never retrieved.","correct":true},{"text":"nDCG@10 over the retriever, since it captures ranking quality better than any single-cutoff metric.","correct":false,"tempting":"Reaches for the most sophisticated ranking metric when the binding question is simply whether the answer was present in what the model was given."},{"text":"Precision@5, since irrelevant passages in the context distract the model and lower answer quality.","correct":false,"tempting":"Optimises for context tidiness, a second-order effect, while the first-order failure of a missing passage goes unmeasured."},{"text":"Judge agreement on the 58% figure, since the answer-correctness labels may themselves be unreliable.","correct":false,"tempting":"Questions the measuring instrument, which is worth doing eventually, but leaves the pipeline's two halves still fused into one unusable number."}]},"prev":{"slug":"metrics-for-numbers","title":"When the output is a quantity","url":"https://addaly.com/learn/evaluating-ai/metrics-for-numbers"},"next":{"slug":"evaluating-rag","title":"Three questions, not one, for a retrieval system","url":"https://addaly.com/learn/evaluating-ai/evaluating-rag"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Scoring a ranked list', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/retrieval-metrics (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Scoring a ranked list","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"For a retrieval-augmented system, measure recall at the number of passages you actually pass to the model, because the generator cannot recover an answer that was never retrieved.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"when-the-output-is-an-ordered-list","runs":[{"text":"When the output is an ordered list"}]},{"t":"p","runs":[{"text":"Search, recommendation and the retrieval half of any RAG system all produce the same shape of output: a ranked list, of which the user or the model downstream sees the top few. Precision and recall need adapting, because position matters: the right document at rank 1 and the right document at rank 9 are not the same result."}]},{"t":"p","runs":[{"text":"Set up the evaluation the same way as everything else in this course. Fifty to a hundred real questions, and for each one, the identifiers of the documents that contain the answer. Building that mapping is the work; once you have it, every metric below is arithmetic."}]},{"t":"p","runs":[{"text":"A note on collecting it: do not write the questions by reading your documents. You will write questions shaped like the text, and the system will look better than it is. Take real questions (from search logs, support tickets, the sales inbox) and then find which document answers each one."}]},{"t":"h2","id":"the-four-metrics-worked","runs":[{"text":"The four metrics, worked"}]},{"t":"p","runs":[{"text":"A query with two relevant documents, "},{"text":"D3","code":true},{"text":" and "},{"text":"D7","code":true},{"text":". The system returns:"}]},{"t":"code","lang":"","label":"Text","text":"rank 1: D9   (not relevant)\nrank 2: D3   (relevant)\nrank 3: D5   (not relevant)\nrank 4: D2   (not relevant)\nrank 5: D7   (relevant)"},{"t":"p","runs":[{"text":"Recall@k","b":true},{"text":": what fraction of the relevant documents appear in the top k?"}]},{"t":"code","lang":"","label":"Text","text":"recall@3 = 1/2 = 0.50\nrecall@5 = 2/2 = 1.00"},{"t":"p","runs":[{"text":"Precision@k","b":true},{"text":": what fraction of the top k are relevant?"}]},{"t":"code","lang":"","label":"Text","text":"precision@3 = 1/3 = 0.33\nprecision@5 = 2/5 = 0.40"},{"t":"p","runs":[{"text":"MRR","b":true},{"text":": reciprocal rank of the "},{"text":"first","i":true},{"text":" relevant result, averaged across queries. Here the first relevant hit is at rank 2, so "},{"text":"1/2 = 0.5","code":true},{"text":". Use MRR when the user needs one good answer and will stop reading once they find it."}]},{"t":"p","runs":[{"text":"nDCG@k","b":true},{"text":": discounted cumulative gain, normalised. It gives credit that falls off with position ("},{"text":"1/log2(rank+1)","code":true},{"text":") and, unlike the others, handles graded relevance: a document can be perfect, partly useful or useless rather than only relevant or not. Use it when relevance is a matter of degree, and be aware that it is the metric people quote most and inspect least."}]},{"t":"h2","id":"for-rag-recall-k-is-the-one-that-matters","runs":[{"text":"For RAG, recall@k is the one that matters"}]},{"t":"p","runs":[{"text":"Here is the practical instruction. If you are retrieving passages to put into a model’s context, measure "},{"text":"recall@k where k is the number of passages you actually pass in","b":true},{"text":". Nothing else comes close in importance."}]},{"t":"p","runs":[{"text":"The reason is a hard constraint. If the answer is not in the retrieved context, no prompt, no model upgrade and no clever instruction can produce a correct grounded answer. The generation step cannot recover information that was never handed to it. It can only decline, or invent."}]},{"t":"p","runs":[{"text":"Teams routinely spend three weeks rewriting prompts to fix a system whose retrieval recall@5 is 0.55. Measure it first. It takes an afternoon, and about half the time it ends the investigation on the spot."}]},{"t":"p","runs":[{"text":"Ordering "},{"text":"within","i":true},{"text":" the retrieved set matters less than people expect for RAG, because the model reads all of it, though not evenly: content in the middle of a long context is used less reliably than content at either end. Keep k modest and put the strongest passage first."}]},{"t":"h2","id":"where-retrieval-fails","runs":[{"text":"Where retrieval fails"}]},{"t":"p","runs":[{"text":"When recall@5 is poor, the cause is usually one of four things, and they are distinguishable by reading twenty failed queries:"}]},{"t":"ul","items":[[{"text":"Chunking.","b":true},{"text":" The answer spans a boundary, so no single chunk contains it. Overlapping windows, or chunking on document structure rather than a fixed character count, fixes more retrieval problems than any embedding-model change."}],[{"text":"Vocabulary mismatch.","b":true},{"text":" The user says “can’t log in”, the document says “authentication failure”. Dense embeddings help here and are exactly why they exist, but they are weak in the opposite case."}],[{"text":"Exact tokens.","b":true},{"text":" Part numbers, error codes, names, dates. Embeddings blur these; keyword search does not miss them. Hybrid retrieval (BM25 and dense vectors, results merged with reciprocal rank fusion) beats either alone on most real corpora, and BM25 is free in "},{"text":"rank_bm25","code":true},{"text":", SQLite FTS5 or Postgres full-text search."}],[{"text":"The document does not exist.","b":true},{"text":" Roughly one failure in five, in my experience of looking, is a question your corpus cannot answer. That is a content problem, and no retrieval work will fix it. It also means your system needs a correct way to say so, which is the subject of the next lesson."}]]},{"t":"h2","id":"tools","runs":[{"text":"Tools"}]},{"t":"p","runs":[{"text":"ranx","code":true},{"text":" and "},{"text":"pytrec_eval","code":true},{"text":" compute all four metrics from a run file and are free and open source. BEIR is a standard suite of retrieval benchmarks, useful for choosing an embedding model to start from, useless as a substitute for measuring on your own corpus. Honestly, twenty lines of Python over a dictionary of "},{"text":"{query_id: [relevant_doc_ids]}","code":true},{"text":" will do everything in this lesson, and writing it yourself means you know what the numbers mean."}]}],"check":{"question":[{"text":"A RAG assistant answers 58% of questions correctly. The team is comparing three prompt rewrites to improve it. What should they measure first instead?"}],"options":[{"text":[{"text":"Recall@k at the number of passages fed to the model, since a correct answer is impossible when the supporting passage was never retrieved."}],"correct":true,"tempting":null},{"text":[{"text":"nDCG@10 over the retriever, since it captures ranking quality better than any single-cutoff metric."}],"correct":false,"tempting":[{"text":"Reaches for the most sophisticated ranking metric when the binding question is simply whether the answer was present in what the model was given."}]},{"text":[{"text":"Precision@5, since irrelevant passages in the context distract the model and lower answer quality."}],"correct":false,"tempting":[{"text":"Optimises for context tidiness, a second-order effect, while the first-order failure of a missing passage goes unmeasured."}]},{"text":[{"text":"Judge agreement on the 58% figure, since the answer-correctness labels may themselves be unreliable."}],"correct":false,"tempting":[{"text":"Questions the measuring instrument, which is worth doing eventually, but leaves the pipeline’s two halves still fused into one unusable number."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"metrics-for-numbers","title":"When the output is a quantity","minutes":9},"next":{"slug":"evaluating-rag","title":"Three questions, not one, for a retrieval system","minutes":10}}}