{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-rag","title":"Three questions, not one, for a retrieval system","url":"https://addaly.com/learn/evaluating-ai/evaluating-rag","position":19,"of":76,"minutes":10,"answer":"Measure retrieval, groundedness and relevance separately, and always include questions your corpus cannot answer so you find out whether the system says so.","keyIdea":"Measure retrieval, groundedness and relevance separately, and always include questions your corpus cannot answer so you find out whether the system says so."},"body":{"format":"markdown","text":"## One number cannot diagnose a two-stage pipeline\n\nA retrieval-augmented system does two separable things: it finds passages, then it writes an answer from them. \"Answer correctness\" fuses the two, so a fall from 71% to 62% tells you a great deal about your morning and nothing about where to look.\n\nSplit it into three measurements, each with its own number:\n\n1. **Retrieval.** Was the supporting passage in the context? Recall@k, from the previous lesson.\n2. **Groundedness.** Is every factual claim in the answer supported by the passages that were supplied? Also called faithfulness.\n3. **Relevance.** Does the answer address what was actually asked?\n\nThe combinations are diagnostic, and each points somewhere different:\n\n- Retrieval fails, answer wrong: fix chunking and search. Nothing else will help.\n- Retrieval succeeds, groundedness low: the model is adding material. Tighten the prompt, lower temperature, require quotation.\n- Retrieval succeeds, groundedness high, answer still unhelpful: the passage is in the corpus but does not really answer the question. A content problem.\n- Retrieval fails and the answer is confidently wrong: the worst quadrant, and the one nobody measures. Cover it below.\n\n## Measuring groundedness\n\nGroundedness is checkable mechanically, which makes it one of the few LLM qualities you can evaluate cheaply and reliably.\n\nThe method: break the answer into atomic claims, then check each claim against the supplied context on its own.\n\n```\nStep 1 — decompose\nInput: the answer text.\nOutput: a JSON list of standalone factual claims,\n        each understandable without the others.\n\nStep 2 — verify, one call per claim\nCONTEXT: {retrieved passages}\nCLAIM:   {one claim}\n\nQuote the span of CONTEXT that supports the claim, or write\nNONE. Then output {\"quote\": \"...\", \"supported\": true|false}.\n\ngroundedness = supported_claims / total_claims\n```\n\nOne claim per call matters. Ask a model to check six claims at once and its errors correlate: it decides the answer \"looks right\" and waves all six through. Ask separately and you get six independent judgements, which is the whole point.\n\nRequiring a quoted span, rather than a yes/no, is the other half. A judge that must point at evidence is markedly harder to fool, and you can verify the quote is a genuine substring of the context in one line of code: a free, deterministic check on your judge's honesty.\n\n## The refusal case nobody tests\n\nHere is the most common gap in RAG evaluation, and it is easy to close.\n\nEvery set is built from questions the corpus can answer. So the system is never asked something the documents do not cover, and nobody finds out what it does then. In production it is asked constantly, and what it does is write a fluent, plausible, entirely invented answer.\n\n> Figure: Four numbers for a retrieval-augmented system. Steps: 1. Retrieval (Was the supporting passage in the context? Recall at the k you actually pass in.); 2. Groundedness (Is every claim supported by the supplied passages? One claim per call, quote required.); 3. Relevance (Does the answer address what was asked, rather than a neighbouring question?); 4. Abstention (On the twenty unanswerable items, did it say so? Read beside the refusal rate on answerable ones.). One 'answer correctness' figure fuses all four. Split apart, each failure points at a different fix: chunking and search, the prompt, the corpus, or the refusal behaviour.\n\nAdd 20 items to the set where the correct output is a refusal:\n\n- questions about a product you do not sell\n- questions about a period your documents do not cover (\"what was the 2019 policy?\" when your archive starts in 2022)\n- questions with a false premise baked in (\"why was the Mumbai office closed?\" when it was not)\n- questions that need information the corpus deliberately excludes, such as an individual's salary\n\nScore these separately as **abstention rate**: the fraction where the system said it did not know rather than answering. Report it alongside accuracy, always. A system at 88% answer accuracy and 15% abstention on unanswerable questions is more dangerous than one at 79% and 90%, because the first one lies with a citation attached.\n\nAnd track the mirror failure too: refusing questions the corpus *can* answer. Push abstention up carelessly and you get a system that says \"I don't know\" to everything, which scores beautifully on groundedness and is useless. The two numbers only mean something together.\n\n## Citations must be checked, not trusted\n\nIf your system shows sources, verify them programmatically on every eval run:\n\n- the cited identifier exists in the corpus\n- the cited document was actually in the retrieved context for that query\n- the claim near the citation is supported by *that* document, not merely by some other retrieved one\n\nThe second and third checks catch a specific and common failure: the model attaches a plausible citation to a sentence it wrote from its own parameters. Users trust a citation more than they trust prose, so a wrong citation costs more than a wrong sentence.\n\n## Tools, and what to build yourself\n\n**RAGAS** (Apache 2.0) implements faithfulness, answer relevance and context precision out of the box. **TruLens** and **DeepEval** cover similar ground; **Inspect**, from the UK AI Safety Institute, is MIT-licensed and better suited if you want to write your own scorers. All four are free.\n\nUse one to get moving in an afternoon. But build the decomposition-and-quote checker yourself at some point, because the packaged faithfulness scores are themselves LLM judges you have not calibrated, and module three is about to explain why that matters."},"check":{"question":"A support RAG system scores 91% on answer accuracy across a 120-question eval set. Which addition would most change what the team knows?","options":[{"text":"Increasing the set to 400 questions, to tighten the confidence interval around the 91%.","correct":false,"tempting":"Buys precision on a number whose blind spot is structural, so a tighter interval around the same measurement reveals nothing new."},{"text":"Adding a second judge model and reporting the average of the two accuracy scores.","correct":false,"tempting":"Averages two uncalibrated opinions, which smooths the number without testing anything the set does not already cover."},{"text":"Adding 20 questions the corpus cannot answer, and scoring separately how often the system says so rather than inventing an answer.","correct":true},{"text":"Recording latency and cost per question alongside accuracy, so quality is not optimised in isolation.","correct":false,"tempting":"Adds operational metrics that belong in the report but say nothing about the system's behaviour on questions outside its knowledge."}]},"prev":{"slug":"retrieval-metrics","title":"Scoring a ranked list","url":"https://addaly.com/learn/evaluating-ai/retrieval-metrics"},"next":{"slug":"the-benchmark-you-read-about","title":"Public benchmarks, and what they are for","url":"https://addaly.com/learn/evaluating-ai/the-benchmark-you-read-about"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Three questions, not one, for a retrieval system', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-rag (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Three questions, not one, for a retrieval system","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"Measure retrieval, groundedness and relevance separately, and always include questions your corpus cannot answer so you find out whether the system says so.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Retrieval","detail":"Was the supporting passage in the context? Recall at the k you actually pass in."},{"label":"Groundedness","detail":"Is every claim supported by the supplied passages? One claim per call, quote required."},{"label":"Relevance","detail":"Does the answer address what was asked, rather than a neighbouring question?"},{"label":"Abstention","detail":"On the twenty unanswerable items, did it say so? Read beside the refusal rate on answerable ones."}],"title":"Four numbers for a retrieval-augmented system","caption":"One ‘answer correctness’ figure fuses all four. Split apart, each failure points at a different fix: chunking and search, the prompt, the corpus, or the refusal behaviour."},"alt":"Figure: Four numbers for a retrieval-augmented system. Steps: 1. Retrieval (Was the supporting passage in the context? Recall at the k you actually pass in.); 2. Groundedness (Is every claim supported by the supplied passages? One claim per call, quote required.); 3. Relevance (Does the answer address what was asked, rather than a neighbouring question?); 4. Abstention (On the twenty unanswerable items, did it say so? Read beside the refusal rate on answerable ones.). One ‘answer correctness’ figure fuses all four. Split apart, each failure points at a different fix: chunking and search, the prompt, the corpus, or the refusal behaviour."}},"blocks":[{"t":"h2","id":"one-number-cannot-diagnose-a-two-stage-pipeline","runs":[{"text":"One number cannot diagnose a two-stage pipeline"}]},{"t":"p","runs":[{"text":"A retrieval-augmented system does two separable things: it finds passages, then it writes an answer from them. “Answer correctness” fuses the two, so a fall from 71% to 62% tells you a great deal about your morning and nothing about where to look."}]},{"t":"p","runs":[{"text":"Split it into three measurements, each with its own number:"}]},{"t":"ol","start":1,"items":[[{"text":"Retrieval.","b":true},{"text":" Was the supporting passage in the context? Recall@k, from the previous lesson."}],[{"text":"Groundedness.","b":true},{"text":" Is every factual claim in the answer supported by the passages that were supplied? Also called faithfulness."}],[{"text":"Relevance.","b":true},{"text":" Does the answer address what was actually asked?"}]]},{"t":"p","runs":[{"text":"The combinations are diagnostic, and each points somewhere different:"}]},{"t":"ul","items":[[{"text":"Retrieval fails, answer wrong: fix chunking and search. Nothing else will help."}],[{"text":"Retrieval succeeds, groundedness low: the model is adding material. Tighten the prompt, lower temperature, require quotation."}],[{"text":"Retrieval succeeds, groundedness high, answer still unhelpful: the passage is in the corpus but does not really answer the question. A content problem."}],[{"text":"Retrieval fails and the answer is confidently wrong: the worst quadrant, and the one nobody measures. Cover it below."}]]},{"t":"h2","id":"measuring-groundedness","runs":[{"text":"Measuring groundedness"}]},{"t":"p","runs":[{"text":"Groundedness is checkable mechanically, which makes it one of the few LLM qualities you can evaluate cheaply and reliably."}]},{"t":"p","runs":[{"text":"The method: break the answer into atomic claims, then check each claim against the supplied context on its own."}]},{"t":"code","lang":"","label":"Text","text":"Step 1 — decompose\nInput: the answer text.\nOutput: a JSON list of standalone factual claims,\n        each understandable without the others.\n\nStep 2 — verify, one call per claim\nCONTEXT: {retrieved passages}\nCLAIM:   {one claim}\n\nQuote the span of CONTEXT that supports the claim, or write\nNONE. Then output {\"quote\": \"...\", \"supported\": true|false}.\n\ngroundedness = supported_claims / total_claims"},{"t":"p","runs":[{"text":"One claim per call matters. Ask a model to check six claims at once and its errors correlate: it decides the answer “looks right” and waves all six through. Ask separately and you get six independent judgements, which is the whole point."}]},{"t":"p","runs":[{"text":"Requiring a quoted span, rather than a yes/no, is the other half. A judge that must point at evidence is markedly harder to fool, and you can verify the quote is a genuine substring of the context in one line of code: a free, deterministic check on your judge’s honesty."}]},{"t":"h2","id":"the-refusal-case-nobody-tests","runs":[{"text":"The refusal case nobody tests"}]},{"t":"p","runs":[{"text":"Here is the most common gap in RAG evaluation, and it is easy to close."}]},{"t":"p","runs":[{"text":"Every set is built from questions the corpus can answer. So the system is never asked something the documents do not cover, and nobody finds out what it does then. In production it is asked constantly, and what it does is write a fluent, plausible, entirely invented answer."}]},{"t":"p","runs":[{"text":"Add 20 items to the set where the correct output is a refusal:"}]},{"t":"ul","items":[[{"text":"questions about a product you do not sell"}],[{"text":"questions about a period your documents do not cover (“what was the 2019 policy?” when your archive starts in 2022)"}],[{"text":"questions with a false premise baked in (“why was the Mumbai office closed?” when it was not)"}],[{"text":"questions that need information the corpus deliberately excludes, such as an individual’s salary"}]]},{"t":"p","runs":[{"text":"Score these separately as "},{"text":"abstention rate","b":true},{"text":": the fraction where the system said it did not know rather than answering. Report it alongside accuracy, always. A system at 88% answer accuracy and 15% abstention on unanswerable questions is more dangerous than one at 79% and 90%, because the first one lies with a citation attached."}]},{"t":"p","runs":[{"text":"And track the mirror failure too: refusing questions the corpus "},{"text":"can","i":true},{"text":" answer. Push abstention up carelessly and you get a system that says “I don’t know” to everything, which scores beautifully on groundedness and is useless. The two numbers only mean something together."}]},{"t":"h2","id":"citations-must-be-checked-not-trusted","runs":[{"text":"Citations must be checked, not trusted"}]},{"t":"p","runs":[{"text":"If your system shows sources, verify them programmatically on every eval run:"}]},{"t":"ul","items":[[{"text":"the cited identifier exists in the corpus"}],[{"text":"the cited document was actually in the retrieved context for that query"}],[{"text":"the claim near the citation is supported by "},{"text":"that","i":true},{"text":" document, not merely by some other retrieved one"}]]},{"t":"p","runs":[{"text":"The second and third checks catch a specific and common failure: the model attaches a plausible citation to a sentence it wrote from its own parameters. Users trust a citation more than they trust prose, so a wrong citation costs more than a wrong sentence."}]},{"t":"h2","id":"tools-and-what-to-build-yourself","runs":[{"text":"Tools, and what to build yourself"}]},{"t":"p","runs":[{"text":"RAGAS","b":true},{"text":" (Apache 2.0) implements faithfulness, answer relevance and context precision out of the box. "},{"text":"TruLens","b":true},{"text":" and "},{"text":"DeepEval","b":true},{"text":" cover similar ground; "},{"text":"Inspect","b":true},{"text":", from the UK AI Safety Institute, is MIT-licensed and better suited if you want to write your own scorers. All four are free."}]},{"t":"p","runs":[{"text":"Use one to get moving in an afternoon. But build the decomposition-and-quote checker yourself at some point, because the packaged faithfulness scores are themselves LLM judges you have not calibrated, and module three is about to explain why that matters."}]}],"check":{"question":[{"text":"A support RAG system scores 91% on answer accuracy across a 120-question eval set. Which addition would most change what the team knows?"}],"options":[{"text":[{"text":"Increasing the set to 400 questions, to tighten the confidence interval around the 91%."}],"correct":false,"tempting":[{"text":"Buys precision on a number whose blind spot is structural, so a tighter interval around the same measurement reveals nothing new."}]},{"text":[{"text":"Adding a second judge model and reporting the average of the two accuracy scores."}],"correct":false,"tempting":[{"text":"Averages two uncalibrated opinions, which smooths the number without testing anything the set does not already cover."}]},{"text":[{"text":"Adding 20 questions the corpus cannot answer, and scoring separately how often the system says so rather than inventing an answer."}],"correct":true,"tempting":null},{"text":[{"text":"Recording latency and cost per question alongside accuracy, so quality is not optimised in isolation."}],"correct":false,"tempting":[{"text":"Adds operational metrics that belong in the report but say nothing about the system’s behaviour on questions outside its knowledge."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"retrieval-metrics","title":"Scoring a ranked list","minutes":9},"next":{"slug":"the-benchmark-you-read-about","title":"Public benchmarks, and what they are for","minutes":8}}}