Measuring retrieval on its own

Two stages, one wrong answer

A retrieval feature gives a wrong answer. The prompt is blamed, tuned, and the answer is still wrong, because the right passage was never retrieved and no prompt can make the model cite what it did not see. Or the passage was retrieved, and the model ignored it, and a week is spent on the embedding model instead. Every wrong answer from a retrieval-augmented system has two candidate causes, and the only way to tell them apart is to measure the retrieval stage separately from the generation stage.

The gold set

Forty questions, each paired with the ids of the chunks that answer it. Not the answer text: the chunk ids. That is the gold set for retrieval, and it is built by hand: take real questions from the logs and the spec, search the corpus yourself, and record which chunks contain the answer. Some questions have one answering chunk; some have three; a few have none, and those are kept, tagged unanswerable, because the correct retrieval result for them is an empty or low-scored list.

It takes an afternoon. Store it beside the case file from the earlier lesson, with the same provenance field, because chunk ids change when the corpus is re-ingested and you need to know which version of the index the set was built against.

The two numbers

Recall at k: for each question, is at least one of its gold chunks in the top k results? Averaged over the set. At k equal to the number of chunks you send to the model, this is the fraction of questions for which the model was given the answer. If it is 0.7, three questions in ten were unanswerable before generation began, whatever the prompt.

Mean reciprocal rank: for each question, one divided by the rank of the first gold chunk, averaged. A system that puts the right chunk first every time scores 1.0; one that puts it fifth scores 0.2 on that question. It tells you whether the right chunk is arriving at the top, where the model attends to it, or at the bottom, where the reranking lesson showed it is often ignored.

def recall_at_k(results, gold, k):
    return sum(any(c in gold[q] for c in results[q][:k]) for q in gold) / len(gold)

def mrr(results, gold):
    total = 0
    for q in gold:
        ranks = [i for i, c in enumerate(results[q], 1) if c in gold[q]]
        total += 1 / ranks[0] if ranks else 0
    return total / len(gold)

That is the whole measurement. The ranx library does it with more metrics and less code, if you prefer.

The two-by-two that ends the argument

For every failed case in the end-to-end evaluation, record two facts: was a gold chunk retrieved, and was the answer correct. Four cells.

• Retrieved and correct: fine.
• Not retrieved and wrong: a retrieval failure. Work on chunking, embeddings, hybrid search, rewriting: the retrieval module.
• Retrieved and wrong: a generation failure. The model had the answer and did not use it: work on the prompt, the number of chunks, their order, the model.
• Not retrieved and correct: the model answered from its own knowledge. Convenient today; a hallucination tomorrow when the corpus and the world disagree.

Count the cells. The largest one is where the next week goes. Teams that skip this step tune the wrong stage more often than not, because the prompt is the easiest thing to change and so it is changed first.

What to run it on

Run the retrieval measurement on every change to the retrieval stage (chunk size, embedding model, index parameters, hybrid weights, reranker, query rewriting) before touching the end-to-end evaluation. It is fast: no generation, forty searches, a second of compute. A change that raises recall at five from 0.72 to 0.85 is worth shipping before you know what it does to the final answer, because the final answer cannot get worse from the model being handed the right passage more often.

The set goes stale

Chunk ids change on re-ingestion; documents are edited; the right answer moves. Re-validate the gold set when the corpus changes materially, and store the index version it was built against. A recall number computed against ids that no longer exist is zero, and it will look like a catastrophic regression rather than a stale file.

Unanswerable questions

The unanswerable cases are where a threshold is set. For those questions, the top result’s score (from the reranker, ideally) should sit below the cutoff that triggers “I could not find this”, and for the answerable ones it should sit above. Plot the two distributions; the gap, if there is one, is where the cutoff goes. If there is no gap, your system cannot tell when it has nothing, and it will answer confidently from the wrong passage. Knowing that is worth the afternoon on its own.