Thirty questions, and the numbers they give you

Build the set first

Everything in this module has a dial on it: model, chunk size, overlap, k, reranker, hybrid weight. Without measurement you are turning dials and trusting your impression of five queries, which is how people end up confidently worse than when they started.

The set is thirty to fifty questions, written by you, each paired with the chunk or document that should answer it.

{"q": "How long does a refund take?", "gold": ["policy-refunds-c3"]}
{"q": "मेरा पार्सल कहाँ है?", "gold": ["delivery-tracking-c1", "delivery-faq-c2"]}

Three rules for writing them:

1. Phrase them as users do, not as the document does. A question that reuses the document’s words tests nothing; keyword search would already have found it.
2. Include the hard cases: a misspelling, a question in the other language, an abbreviation, a question whose answer is spread across two chunks.
3. Include five questions with no answer in the corpus. Your system should return nothing useful for those, and measuring that is how you find out whether it invents an answer.

Store the file in the repository. It is an asset, and it outlives every model you will try.

The three numbers

Recall@k: the fraction of questions whose correct chunk appears in the top k. This measures the retriever alone, and it is the ceiling on everything downstream. If recall@50 is 0.7, no reranker and no language model can answer the other 30%.

Precision@k or hit rate@1: how often the top result is right. This measures ranking, and it is what a reranker improves.

Mean reciprocal rank: the average of 1 divided by the rank of the first correct result. It rewards being right at position one rather than position three, in one number.

def evaluate(search_fn, questions, k=10):
    rec = mrr = 0
    for item in questions:
        got = [d["id"] for d in search_fn(item["q"], k=k)]
        hits = [i for i, d in enumerate(got) if d in item["gold"]]
        if hits:
            rec += 1
            mrr += 1 / (hits[0] + 1)
    n = len(questions)
    return {"recall@k": rec / n, "mrr": round(mrr / n, 3)}

Read the failures, do not only count them

Thirty is small enough to read, and that is the point of choosing thirty rather than three thousand. Print every question that failed alongside what came back instead. The causes sort into a short list, and each has a different fix:

• The gold chunk was never retrieved → chunking or the embedding model.
• It was retrieved at rank 20 → add or improve reranking.
• A near-duplicate outranked it → maximal marginal relevance, or deduplicate the corpus.
• The query used a code or a name → add BM25.
• The answer spans two chunks → larger chunks, more overlap, or parent retrieval.
• The corpus does not contain the answer → not a retrieval problem, and worth knowing before you spend a week on one.

That list is the actual value of the exercise. A number tells you whether something changed; the failures tell you what to change.

What thirty questions can and cannot detect

Thirty items will reliably detect a difference of fifteen or twenty points: a wrong chunk size, a model that does not handle your language, a missing query prefix. It will not detect a difference of two or three points; that is well inside sampling noise, and acting on it is superstition.

So: use thirty to make the big decisions, and do not chase small ones. If a difference matters and looks small, you need several hundred questions, and building those is a different project with a different budget.

Freeze the set, and add to it

Once a set exists, every change is checkable. Keep the original frozen so numbers stay comparable over time, and add a question every time production surprises you. An incident that produced a wrong answer becomes a permanent test item, and the set slowly becomes a record of everything you have learned about your own corpus.

The honest note

This measures retrieval, not answers. A system can retrieve the right chunk and still produce a wrong answer from it, and evaluating generated text is a harder problem with its own course. Measuring retrieval separately is worth doing precisely because it is tractable: when the whole system is wrong, the first question is always whether the right material was even found.

Do this now

Write ten questions in the next fifteen minutes. Not thirty, ten. Run them, read the failures, and fix the one that appears most. That loop, repeated, is the entire discipline.