Measuring retrieval without the model

The number that lets you stop guessing

End-to-end accuracy tells you the system is wrong. It does not tell you which half. Retrieval metrics, measured on their own, split every failure into “the evidence was not there” and “the evidence was there and was not used”, and that split determines which module of this course you should be working in.

It is also cheap. Retrieval scoring needs no model call, so a full run over 500 queries takes seconds and costs nothing. You can put it in CI.

The three metrics, and when each is the right one

Recall@k: in what fraction of queries is at least one relevant passage inside the top k? This is the one that matters most, because it is the ceiling on everything downstream. If recall@10 is 0.72, no reranker, no prompt and no model gets you above 72% on questions needing retrieval.

MRR: the mean of one over the rank of the first relevant passage. Sensitive to whether the right answer is first or fourth. Worth watching when you show a top result directly, less important when you pass ten passages to a model that reads all of them.

nDCG@k: accounts for graded relevance and position. Use it when passages are not simply relevant or irrelevant, for example a search interface where the ordering itself is the product.

For most retrieval-augmented systems, track recall@k at the k you actually use, plus recall at the depth you rerank from. Those two numbers between them tell you whether to work on the first stage or the second.

Building a gold set without a labelling budget

You need queries paired with the passage that answers them. Four sources, best first:

1. Resolved support tickets or answered questions. The question is real and somebody already found the document. This is the highest-quality source and most organisations are sitting on thousands of them without realising they are a test set.

2. Search logs with a click. The query is real; the click is a noisy relevance label. Noisy is fine in aggregate for tracking movement over time.

3. Questions written by people who know the domain. Fifty from a support lead is an afternoon’s work and better than any synthetic set.

4. Model-generated questions from your own chunks. Take a chunk, ask a model to write a question it answers, and you have a pair. Fast, free, scales to thousands.

Option four comes with a caveat you must apply. A question generated from a chunk reuses the chunk’s own vocabulary, so it is unnaturally easy for keyword retrieval and easy for dense retrieval too. Recall measured on a generated set is systematically higher than recall on real user queries, commonly by a wide margin.

Two mitigations. Instruct the generator to write as a user who has not read the document would, using everyday words. And always keep a smaller real set alongside, so you can see how far apart the two numbers are. If your synthetic recall is 0.94 and your real recall is 0.68, quote the second.

Beware the passage that is relevant and not gold

A gold set with one correct passage per query will mark a correct retrieval as a miss whenever the corpus has more than one passage answering the question, which, after the duplication lesson, you know is common.

The practical fix is to allow multiple gold ids per query, and to add them as you find them. When investigating a supposed miss, if the retrieved passage does answer the question, add it to the gold list rather than accepting the false negative. The set improves every time you look at it, and this is one of the few kinds of maintenance that compounds.

The harness

def recall_at(retriever, gold, ks=(1, 5, 10, 50)):
    hits = {k: 0 for k in ks}
    for q, gold_ids in gold:
        ranked = [d.id for d in retriever(q, k=max(ks))]
        for k in ks:
            if set(ranked[:k]) & set(gold_ids):
                hits[k] += 1
    return {f"recall@{k}": hits[k] / len(gold) for k in ks}

Run it on every change to chunking, embeddings, the query rewriter, the fusion, the filters. Each of those is a change to retrieval and each can be checked in seconds without touching a language model.

Slice it, always

One average hides everything interesting. Report recall broken down by query type (identifier lookups, conceptual questions, comparisons) and by language.

That is where hybrid search shows its real value, where a multilingual embedding model’s weakness in one script becomes visible, and where you find that the system works well for the questions you designed for and poorly for the ones people ask. An overall recall of 0.85 made of 0.95 on head queries and 0.55 on the tail is a different system from a uniform 0.85, and only one of the two is ready to ship.