Scoring a ranked list

When the output is an ordered list

Search, recommendation and the retrieval half of any RAG system all produce the same shape of output: a ranked list, of which the user or the model downstream sees the top few. Precision and recall need adapting, because position matters: the right document at rank 1 and the right document at rank 9 are not the same result.

Set up the evaluation the same way as everything else in this course. Fifty to a hundred real questions, and for each one, the identifiers of the documents that contain the answer. Building that mapping is the work; once you have it, every metric below is arithmetic.

A note on collecting it: do not write the questions by reading your documents. You will write questions shaped like the text, and the system will look better than it is. Take real questions (from search logs, support tickets, the sales inbox) and then find which document answers each one.

The four metrics, worked

A query with two relevant documents, D3 and D7. The system returns:

rank 1: D9   (not relevant)
rank 2: D3   (relevant)
rank 3: D5   (not relevant)
rank 4: D2   (not relevant)
rank 5: D7   (relevant)

Recall@k: what fraction of the relevant documents appear in the top k?

recall@3 = 1/2 = 0.50
recall@5 = 2/2 = 1.00

Precision@k: what fraction of the top k are relevant?

precision@3 = 1/3 = 0.33
precision@5 = 2/5 = 0.40

MRR: reciprocal rank of the first relevant result, averaged across queries. Here the first relevant hit is at rank 2, so 1/2 = 0.5. Use MRR when the user needs one good answer and will stop reading once they find it.

nDCG@k: discounted cumulative gain, normalised. It gives credit that falls off with position (1/log2(rank+1)) and, unlike the others, handles graded relevance: a document can be perfect, partly useful or useless rather than only relevant or not. Use it when relevance is a matter of degree, and be aware that it is the metric people quote most and inspect least.

For RAG, recall@k is the one that matters

Here is the practical instruction. If you are retrieving passages to put into a model’s context, measure recall@k where k is the number of passages you actually pass in. Nothing else comes close in importance.

The reason is a hard constraint. If the answer is not in the retrieved context, no prompt, no model upgrade and no clever instruction can produce a correct grounded answer. The generation step cannot recover information that was never handed to it. It can only decline, or invent.

Teams routinely spend three weeks rewriting prompts to fix a system whose retrieval recall@5 is 0.55. Measure it first. It takes an afternoon, and about half the time it ends the investigation on the spot.

Ordering within the retrieved set matters less than people expect for RAG, because the model reads all of it, though not evenly: content in the middle of a long context is used less reliably than content at either end. Keep k modest and put the strongest passage first.

Where retrieval fails

When recall@5 is poor, the cause is usually one of four things, and they are distinguishable by reading twenty failed queries:

• Chunking. The answer spans a boundary, so no single chunk contains it. Overlapping windows, or chunking on document structure rather than a fixed character count, fixes more retrieval problems than any embedding-model change.
• Vocabulary mismatch. The user says “can’t log in”, the document says “authentication failure”. Dense embeddings help here and are exactly why they exist, but they are weak in the opposite case.
• Exact tokens. Part numbers, error codes, names, dates. Embeddings blur these; keyword search does not miss them. Hybrid retrieval (BM25 and dense vectors, results merged with reciprocal rank fusion) beats either alone on most real corpora, and BM25 is free in rank_bm25, SQLite FTS5 or Postgres full-text search.
• The document does not exist. Roughly one failure in five, in my experience of looking, is a question your corpus cannot answer. That is a content problem, and no retrieval work will fix it. It also means your system needs a correct way to say so, which is the subject of the next lesson.

Tools

ranx and pytrec_eval compute all four metrics from a run file and are free and open source. BEIR is a standard suite of retrieval benchmarks, useful for choosing an embedding model to start from, useless as a substitute for measuring on your own corpus. Honestly, twenty lines of Python over a dictionary of {query_id: [relevant_doc_ids]} will do everything in this lesson, and writing it yourself means you know what the numbers mean.