Three questions, not one, for a retrieval system

One number cannot diagnose a two-stage pipeline

A retrieval-augmented system does two separable things: it finds passages, then it writes an answer from them. “Answer correctness” fuses the two, so a fall from 71% to 62% tells you a great deal about your morning and nothing about where to look.

Split it into three measurements, each with its own number:

1. Retrieval. Was the supporting passage in the context? Recall@k, from the previous lesson.
2. Groundedness. Is every factual claim in the answer supported by the passages that were supplied? Also called faithfulness.
3. Relevance. Does the answer address what was actually asked?

The combinations are diagnostic, and each points somewhere different:

• Retrieval fails, answer wrong: fix chunking and search. Nothing else will help.
• Retrieval succeeds, groundedness low: the model is adding material. Tighten the prompt, lower temperature, require quotation.
• Retrieval succeeds, groundedness high, answer still unhelpful: the passage is in the corpus but does not really answer the question. A content problem.
• Retrieval fails and the answer is confidently wrong: the worst quadrant, and the one nobody measures. Cover it below.

Measuring groundedness

Groundedness is checkable mechanically, which makes it one of the few LLM qualities you can evaluate cheaply and reliably.

The method: break the answer into atomic claims, then check each claim against the supplied context on its own.

Step 1 — decompose
Input: the answer text.
Output: a JSON list of standalone factual claims,
        each understandable without the others.

Step 2 — verify, one call per claim
CONTEXT: {retrieved passages}
CLAIM:   {one claim}

Quote the span of CONTEXT that supports the claim, or write
NONE. Then output {"quote": "...", "supported": true|false}.

groundedness = supported_claims / total_claims

One claim per call matters. Ask a model to check six claims at once and its errors correlate: it decides the answer “looks right” and waves all six through. Ask separately and you get six independent judgements, which is the whole point.

Requiring a quoted span, rather than a yes/no, is the other half. A judge that must point at evidence is markedly harder to fool, and you can verify the quote is a genuine substring of the context in one line of code: a free, deterministic check on your judge’s honesty.

The refusal case nobody tests

Here is the most common gap in RAG evaluation, and it is easy to close.

Every set is built from questions the corpus can answer. So the system is never asked something the documents do not cover, and nobody finds out what it does then. In production it is asked constantly, and what it does is write a fluent, plausible, entirely invented answer.

Add 20 items to the set where the correct output is a refusal:

• questions about a product you do not sell
• questions about a period your documents do not cover (“what was the 2019 policy?” when your archive starts in 2022)
• questions with a false premise baked in (“why was the Mumbai office closed?” when it was not)
• questions that need information the corpus deliberately excludes, such as an individual’s salary

Score these separately as abstention rate: the fraction where the system said it did not know rather than answering. Report it alongside accuracy, always. A system at 88% answer accuracy and 15% abstention on unanswerable questions is more dangerous than one at 79% and 90%, because the first one lies with a citation attached.

And track the mirror failure too: refusing questions the corpus can answer. Push abstention up carelessly and you get a system that says “I don’t know” to everything, which scores beautifully on groundedness and is useless. The two numbers only mean something together.

Citations must be checked, not trusted

If your system shows sources, verify them programmatically on every eval run:

• the cited identifier exists in the corpus
• the cited document was actually in the retrieved context for that query
• the claim near the citation is supported by that document, not merely by some other retrieved one

The second and third checks catch a specific and common failure: the model attaches a plausible citation to a sentence it wrote from its own parameters. Users trust a citation more than they trust prose, so a wrong citation costs more than a wrong sentence.

Tools, and what to build yourself

RAGAS (Apache 2.0) implements faithfulness, answer relevance and context precision out of the box. TruLens and DeepEval cover similar ground; Inspect, from the UK AI Safety Institute, is MIT-licensed and better suited if you want to write your own scorers. All four are free.

Use one to get moving in an afternoon. But build the decomposition-and-quote checker yourself at some point, because the packaged faithfulness scores are themselves LLM judges you have not calibrated, and module three is about to explain why that matters.