{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"measuring-your-retrieval","title":"Thirty questions, and the numbers they give you","url":"https://addaly.com/learn/hugging-face/measuring-your-retrieval","position":56,"of":75,"minutes":9,"answer":"Thirty questions of your own with known correct chunks turn every dial in a retrieval system into a measurable decision, and the failures sort into a short list of distinct causes.","keyIdea":"Thirty questions of your own with known correct chunks turn every dial in a retrieval system into a measurable decision, and the failures sort into a short list of distinct causes, but a set that size detects fifteen-point differences and not three-point ones, so it should settle big choices and never small ones."},"body":{"format":"markdown","text":"## Build the set first\n\nEverything in this module has a dial on it: model, chunk size, overlap, k, reranker, hybrid weight. Without measurement you are turning dials and trusting your impression of five queries, which is how people end up confidently worse than when they started.\n\nThe set is thirty to fifty questions, written by you, each paired with the chunk or document that should answer it.\n\n```json\n{\"q\": \"How long does a refund take?\", \"gold\": [\"policy-refunds-c3\"]}\n{\"q\": \"मेरा पार्सल कहाँ है?\", \"gold\": [\"delivery-tracking-c1\", \"delivery-faq-c2\"]}\n```\n\nThree rules for writing them:\n\n1. **Phrase them as users do**, not as the document does. A question that reuses the document's words tests nothing; keyword search would already have found it.\n2. **Include the hard cases**: a misspelling, a question in the other language, an abbreviation, a question whose answer is spread across two chunks.\n3. **Include five questions with no answer in the corpus.** Your system should return nothing useful for those, and measuring that is how you find out whether it invents an answer.\n\nStore the file in the repository. It is an asset, and it outlives every model you will try.\n\n> Figure: What thirty questions can settle, and what they cannot. Detected reliably: A chunk size that is wrong for your documents; An embedding model that does not handle your language; A missing query prefix on a BGE or E5 family model; A reranker's effect, which is usually ten points or more; A corpus that simply does not contain the answer. Not detected: do not act on it: Two points between two shortlisted models; Fifty candidates against sixty; An overlap of forty tokens against fifty. Thirty is chosen because it is small enough to read every failure and large enough to detect the differences worth acting on. Treating twenty-six out of thirty against twenty-seven out of thirty as a result is how people talk themselves into noise.\n\n## The three numbers\n\n**Recall@k**: the fraction of questions whose correct chunk appears in the top k. This measures the retriever alone, and it is the ceiling on everything downstream. If recall@50 is 0.7, no reranker and no language model can answer the other 30%.\n\n**Precision@k or hit rate@1**: how often the top result is right. This measures ranking, and it is what a reranker improves.\n\n**Mean reciprocal rank**: the average of 1 divided by the rank of the first correct result. It rewards being right at position one rather than position three, in one number.\n\n```python\ndef evaluate(search_fn, questions, k=10):\n    rec = mrr = 0\n    for item in questions:\n        got = [d[\"id\"] for d in search_fn(item[\"q\"], k=k)]\n        hits = [i for i, d in enumerate(got) if d in item[\"gold\"]]\n        if hits:\n            rec += 1\n            mrr += 1 / (hits[0] + 1)\n    n = len(questions)\n    return {\"recall@k\": rec / n, \"mrr\": round(mrr / n, 3)}\n```\n\n## Read the failures, do not only count them\n\nThirty is small enough to read, and that is the point of choosing thirty rather than three thousand. Print every question that failed alongside what came back instead. The causes sort into a short list, and each has a different fix:\n\n- **The gold chunk was never retrieved** → chunking or the embedding model.\n- **It was retrieved at rank 20** → add or improve reranking.\n- **A near-duplicate outranked it** → maximal marginal relevance, or deduplicate the corpus.\n- **The query used a code or a name** → add BM25.\n- **The answer spans two chunks** → larger chunks, more overlap, or parent retrieval.\n- **The corpus does not contain the answer** → not a retrieval problem, and worth knowing before you spend a week on one.\n\nThat list is the actual value of the exercise. A number tells you whether something changed; the failures tell you what to change.\n\n## What thirty questions can and cannot detect\n\nThirty items will reliably detect a difference of fifteen or twenty points: a wrong chunk size, a model that does not handle your language, a missing query prefix. It will **not** detect a difference of two or three points; that is well inside sampling noise, and acting on it is superstition.\n\nSo: use thirty to make the big decisions, and do not chase small ones. If a difference matters and looks small, you need several hundred questions, and building those is a different project with a different budget.\n\n## Freeze the set, and add to it\n\nOnce a set exists, every change is checkable. Keep the original frozen so numbers stay comparable over time, and add a question every time production surprises you. An incident that produced a wrong answer becomes a permanent test item, and the set slowly becomes a record of everything you have learned about your own corpus.\n\n## The honest note\n\nThis measures retrieval, not answers. A system can retrieve the right chunk and still produce a wrong answer from it, and evaluating generated text is a harder problem with its own course. Measuring retrieval separately is worth doing precisely because it is tractable: when the whole system is wrong, the first question is always whether the right material was even found.\n\n## Do this now\n\nWrite ten questions in the next fifteen minutes. Not thirty, ten. Run them, read the failures, and fix the one that appears most. That loop, repeated, is the entire discipline."},"check":{"question":"An evaluation shows recall@50 of 0.68 and hit rate@1 of 0.61. Which change is most likely to help?","options":[{"text":"Adding a cross-encoder reranker, since the gap between the two numbers shows poor ranking within the retrieved set.","correct":false,"tempting":"Reranking addresses the gap between recall and precision, which here is small; the binding constraint is the recall ceiling."},{"text":"Retrieving 200 candidates instead of 50, so that more of the corpus is considered before ranking.","correct":false,"tempting":"May raise recall slightly, but a ceiling this low usually means relevant chunks are not findable at any k."},{"text":"Revisiting chunking and the embedding model, because a third of the answers are never retrieved at all.","correct":true},{"text":"Applying maximal marginal relevance, since near-duplicates are crowding out the correct chunk.","correct":false,"tempting":"Diversity fixes crowding within retrieved results and cannot surface a chunk that never appeared."}]},"prev":{"slug":"embedding-images-and-audio","title":"One space for pictures and words","url":"https://addaly.com/learn/hugging-face/embedding-images-and-audio"},"next":{"slug":"when-not-to-fine-tune","title":"Three cheaper things to try first","url":"https://addaly.com/learn/hugging-face/when-not-to-fine-tune"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Thirty questions, and the numbers they give you', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/measuring-your-retrieval (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Thirty questions, and the numbers they give you","courseTitle":"Hugging Face, End to End","module":{"index":6,"slug":"embeddings-and-retrieval","title":"Embeddings, and search that understands"},"answer":null,"keyIdea":"Thirty questions of your own with known correct chunks turn every dial in a retrieval system into a measurable decision, and the failures sort into a short list of distinct causes, but a set that size detects fifteen-point differences and not three-point ones, so it should settle big choices and never small ones.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Detected reliably","items":["A chunk size that is wrong for your documents","An embedding model that does not handle your language","A missing query prefix on a BGE or E5 family model","A reranker’s effect, which is usually ten points or more","A corpus that simply does not contain the answer"]},"type":"compare","right":{"title":"Not detected: do not act on it","items":["Two points between two shortlisted models","Fifty candidates against sixty","An overlap of forty tokens against fifty"]},"title":"What thirty questions can settle, and what they cannot","caption":"Thirty is chosen because it is small enough to read every failure and large enough to detect the differences worth acting on. Treating twenty-six out of thirty against twenty-seven out of thirty as a result is how people talk themselves into noise."},"alt":"Figure: What thirty questions can settle, and what they cannot. Detected reliably: A chunk size that is wrong for your documents; An embedding model that does not handle your language; A missing query prefix on a BGE or E5 family model; A reranker’s effect, which is usually ten points or more; A corpus that simply does not contain the answer. Not detected: do not act on it: Two points between two shortlisted models; Fifty candidates against sixty; An overlap of forty tokens against fifty. Thirty is chosen because it is small enough to read every failure and large enough to detect the differences worth acting on. Treating twenty-six out of thirty against twenty-seven out of thirty as a result is how people talk themselves into noise."}},"blocks":[{"t":"h2","id":"build-the-set-first","runs":[{"text":"Build the set first"}]},{"t":"p","runs":[{"text":"Everything in this module has a dial on it: model, chunk size, overlap, k, reranker, hybrid weight. Without measurement you are turning dials and trusting your impression of five queries, which is how people end up confidently worse than when they started."}]},{"t":"p","runs":[{"text":"The set is thirty to fifty questions, written by you, each paired with the chunk or document that should answer it."}]},{"t":"code","lang":"json","label":"JSON","text":"{\"q\": \"How long does a refund take?\", \"gold\": [\"policy-refunds-c3\"]}\n{\"q\": \"मेरा पार्सल कहाँ है?\", \"gold\": [\"delivery-tracking-c1\", \"delivery-faq-c2\"]}"},{"t":"p","runs":[{"text":"Three rules for writing them:"}]},{"t":"ol","start":1,"items":[[{"text":"Phrase them as users do","b":true},{"text":", not as the document does. A question that reuses the document’s words tests nothing; keyword search would already have found it."}],[{"text":"Include the hard cases","b":true},{"text":": a misspelling, a question in the other language, an abbreviation, a question whose answer is spread across two chunks."}],[{"text":"Include five questions with no answer in the corpus.","b":true},{"text":" Your system should return nothing useful for those, and measuring that is how you find out whether it invents an answer."}]]},{"t":"p","runs":[{"text":"Store the file in the repository. It is an asset, and it outlives every model you will try."}]},{"t":"h2","id":"the-three-numbers","runs":[{"text":"The three numbers"}]},{"t":"p","runs":[{"text":"Recall@k","b":true},{"text":": the fraction of questions whose correct chunk appears in the top k. This measures the retriever alone, and it is the ceiling on everything downstream. If recall@50 is 0.7, no reranker and no language model can answer the other 30%."}]},{"t":"p","runs":[{"text":"Precision@k or hit rate@1","b":true},{"text":": how often the top result is right. This measures ranking, and it is what a reranker improves."}]},{"t":"p","runs":[{"text":"Mean reciprocal rank","b":true},{"text":": the average of 1 divided by the rank of the first correct result. It rewards being right at position one rather than position three, in one number."}]},{"t":"code","lang":"python","label":"Python","text":"def evaluate(search_fn, questions, k=10):\n    rec = mrr = 0\n    for item in questions:\n        got = [d[\"id\"] for d in search_fn(item[\"q\"], k=k)]\n        hits = [i for i, d in enumerate(got) if d in item[\"gold\"]]\n        if hits:\n            rec += 1\n            mrr += 1 / (hits[0] + 1)\n    n = len(questions)\n    return {\"recall@k\": rec / n, \"mrr\": round(mrr / n, 3)}"},{"t":"h2","id":"read-the-failures-do-not-only-count-them","runs":[{"text":"Read the failures, do not only count them"}]},{"t":"p","runs":[{"text":"Thirty is small enough to read, and that is the point of choosing thirty rather than three thousand. Print every question that failed alongside what came back instead. The causes sort into a short list, and each has a different fix:"}]},{"t":"ul","items":[[{"text":"The gold chunk was never retrieved","b":true},{"text":" → chunking or the embedding model."}],[{"text":"It was retrieved at rank 20","b":true},{"text":" → add or improve reranking."}],[{"text":"A near-duplicate outranked it","b":true},{"text":" → maximal marginal relevance, or deduplicate the corpus."}],[{"text":"The query used a code or a name","b":true},{"text":" → add BM25."}],[{"text":"The answer spans two chunks","b":true},{"text":" → larger chunks, more overlap, or parent retrieval."}],[{"text":"The corpus does not contain the answer","b":true},{"text":" → not a retrieval problem, and worth knowing before you spend a week on one."}]]},{"t":"p","runs":[{"text":"That list is the actual value of the exercise. A number tells you whether something changed; the failures tell you what to change."}]},{"t":"h2","id":"what-thirty-questions-can-and-cannot-detect","runs":[{"text":"What thirty questions can and cannot detect"}]},{"t":"p","runs":[{"text":"Thirty items will reliably detect a difference of fifteen or twenty points: a wrong chunk size, a model that does not handle your language, a missing query prefix. It will "},{"text":"not","b":true},{"text":" detect a difference of two or three points; that is well inside sampling noise, and acting on it is superstition."}]},{"t":"p","runs":[{"text":"So: use thirty to make the big decisions, and do not chase small ones. If a difference matters and looks small, you need several hundred questions, and building those is a different project with a different budget."}]},{"t":"h2","id":"freeze-the-set-and-add-to-it","runs":[{"text":"Freeze the set, and add to it"}]},{"t":"p","runs":[{"text":"Once a set exists, every change is checkable. Keep the original frozen so numbers stay comparable over time, and add a question every time production surprises you. An incident that produced a wrong answer becomes a permanent test item, and the set slowly becomes a record of everything you have learned about your own corpus."}]},{"t":"h2","id":"the-honest-note","runs":[{"text":"The honest note"}]},{"t":"p","runs":[{"text":"This measures retrieval, not answers. A system can retrieve the right chunk and still produce a wrong answer from it, and evaluating generated text is a harder problem with its own course. Measuring retrieval separately is worth doing precisely because it is tractable: when the whole system is wrong, the first question is always whether the right material was even found."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Write ten questions in the next fifteen minutes. Not thirty, ten. Run them, read the failures, and fix the one that appears most. That loop, repeated, is the entire discipline."}]}],"check":{"question":[{"text":"An evaluation shows recall@50 of 0.68 and hit rate@1 of 0.61. Which change is most likely to help?"}],"options":[{"text":[{"text":"Adding a cross-encoder reranker, since the gap between the two numbers shows poor ranking within the retrieved set."}],"correct":false,"tempting":[{"text":"Reranking addresses the gap between recall and precision, which here is small; the binding constraint is the recall ceiling."}]},{"text":[{"text":"Retrieving 200 candidates instead of 50, so that more of the corpus is considered before ranking."}],"correct":false,"tempting":[{"text":"May raise recall slightly, but a ceiling this low usually means relevant chunks are not findable at any k."}]},{"text":[{"text":"Revisiting chunking and the embedding model, because a third of the answers are never retrieved at all."}],"correct":true,"tempting":null},{"text":[{"text":"Applying maximal marginal relevance, since near-duplicates are crowding out the correct chunk."}],"correct":false,"tempting":[{"text":"Diversity fixes crowding within retrieved results and cannot surface a chunk that never appeared."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"embedding-images-and-audio","title":"One space for pictures and words","minutes":8},"next":{"slug":"when-not-to-fine-tune","title":"Three cheaper things to try first","minutes":8}}}