Evaluating an embedding model on your corpus

Why the leaderboard will not decide this

MTEB is the standard embedding benchmark and it has two problems that matter here. Models are routinely trained on the training splits of MTEB’s own tasks, so the aggregate increasingly measures familiarity with those datasets. And the tasks are dominated by English and by web-shaped text; if your corpus is support tickets in Hindi, or contracts, or product listings, the ranking transfers weakly.

The good news is that evaluating retrieval on your own corpus is easy (much easier than evaluating a language model) because there is a right answer and it is cheap to check.

Building the set: two hours

1. Take 50 real queries from your logs. If you have no logs, write them by asking three colleagues what they would type, which is worse but workable.
2. For each, find the document or chunk that answers it. Mark it. Where several are relevant, mark them all. This is the only labour, and 50 queries is about ninety minutes.
3. Keep it as JSONL: query, list of relevant chunk identifiers.

That set is now a permanent asset. It survives model changes, chunking changes and reranker changes, and it is the thing that makes every later decision an hour’s work instead of an argument.

The metrics, and which to use

Recall at k: the fraction of queries whose relevant document appears in the top k. This is the number for the retrieval stage. Measure it at 100, because that is what the reranker will receive.

MRR: the mean of one divided by the rank of the first relevant result. Sensitive to the top of the list, appropriate when the user sees one answer.

nDCG at 10: discounted gain, handling multiple relevant documents with graded relevance. The standard for a ranked list somebody reads.

Use recall at 100 for the retriever and nDCG at 10 after reranking. Those two numbers separate the two stages, which is the diagnostic that matters, as the reranker lesson showed.

def recall_at_k(results, relevant, k):
    return sum(bool(set(r[:k]) & set(rel))
               for r, rel in zip(results, relevant)) / len(relevant)

The grid worth running

Chunking usually matters more than the model, and almost nobody tests it. Run the grid:

• Three embedding models: a small one, a base one, a multilingual one.
• Three chunk sizes: for instance 256, 512 and 1,024 tokens, with a small overlap.

Nine configurations, each a few minutes to index on 50,000 chunks with a small model on a CPU. The common finding is that the chunk size moves recall more than the model choice does, and that the best chunk size depends on your documents rather than on any general advice.

While you are there, record for each model: embedding throughput (documents per second on the hardware you will use), dimension, and index size. A model two points better on recall that produces 1024-dimension vectors instead of 384 costs nearly three times the index memory, and that is a real trade rather than a rounding error.

The details that invalidate a comparison

• Prefixes. If one model needs query: and passage: and you did not apply them, you have measured the wrong thing. Check the card for every model in the grid.
• Normalisation. Consistent across all configurations, or the scores are not comparable.
• Truncation. If your chunks exceed a model’s sequence limit, part of every chunk is invisible. This alone can explain a model doing badly.
• The same chunks for every model. Change one variable at a time.

Afterwards

Add a hybrid configuration (dense plus BM25 with rank fusion) and a reranker on top of the winner. Both usually help, and now you can say by how much on your own material rather than quoting somebody’s blog post.

Re-run the whole grid whenever the corpus changes materially. It is one command by then, and a corpus that has doubled in size or gained a new document type is a different retrieval problem from the one you tuned for.