{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"evaluating-an-embedding-model","title":"Evaluating an embedding model on your corpus","url":"https://addaly.com/learn/open-models/evaluating-an-embedding-model","position":67,"of":71,"minutes":9,"answer":"Retrieval is cheap to evaluate on your own corpus with 50 labelled queries, and the grid of embedding model against chunk size usually shows chunking moving recall more than the model does.","keyIdea":"Retrieval is cheap to evaluate on your own corpus with 50 labelled queries, and the grid of embedding model against chunk size usually shows chunking moving recall more than the model does."},"body":{"format":"markdown","text":"## Why the leaderboard will not decide this\n\nMTEB is the standard embedding benchmark and it has two problems that matter here. Models are routinely trained on the training splits of MTEB's own tasks, so the aggregate increasingly measures familiarity with those datasets. And the tasks are dominated by English and by web-shaped text; if your corpus is support tickets in Hindi, or contracts, or product listings, the ranking transfers weakly.\n\nThe good news is that evaluating retrieval on your own corpus is easy (much easier than evaluating a language model) because there is a right answer and it is cheap to check.\n\n## Building the set: two hours\n\n1. **Take 50 real queries** from your logs. If you have no logs, write them by asking three colleagues what they would type, which is worse but workable.\n2. **For each, find the document or chunk that answers it.** Mark it. Where several are relevant, mark them all. This is the only labour, and 50 queries is about ninety minutes.\n3. **Keep it as JSONL**: query, list of relevant chunk identifiers.\n\nThat set is now a permanent asset. It survives model changes, chunking changes and reranker changes, and it is the thing that makes every later decision an hour's work instead of an argument.\n\n## The metrics, and which to use\n\n**Recall at k**: the fraction of queries whose relevant document appears in the top k. This is the number for the retrieval stage. Measure it at 100, because that is what the reranker will receive.\n\n**MRR**: the mean of one divided by the rank of the first relevant result. Sensitive to the top of the list, appropriate when the user sees one answer.\n\n**nDCG at 10**: discounted gain, handling multiple relevant documents with graded relevance. The standard for a ranked list somebody reads.\n\nUse recall at 100 for the retriever and nDCG at 10 after reranking. Those two numbers separate the two stages, which is the diagnostic that matters, as the reranker lesson showed.\n\n```python\ndef recall_at_k(results, relevant, k):\n    return sum(bool(set(r[:k]) & set(rel))\n               for r, rel in zip(results, relevant)) / len(relevant)\n```\n\n## The grid worth running\n\nChunking usually matters more than the model, and almost nobody tests it. Run the grid:\n\n- Three embedding models: a small one, a base one, a multilingual one.\n- Three chunk sizes: for instance 256, 512 and 1,024 tokens, with a small overlap.\n\nNine configurations, each a few minutes to index on 50,000 chunks with a small model on a CPU. The common finding is that the chunk size moves recall more than the model choice does, and that the best chunk size depends on your documents rather than on any general advice.\n\nWhile you are there, record for each model: **embedding throughput** (documents per second on the hardware you will use), **dimension**, and **index size**. A model two points better on recall that produces 1024-dimension vectors instead of 384 costs nearly three times the index memory, and that is a real trade rather than a rounding error.\n\n## The details that invalidate a comparison\n\n- **Prefixes.** If one model needs `query: ` and `passage: ` and you did not apply them, you have measured the wrong thing. Check the card for every model in the grid.\n- **Normalisation.** Consistent across all configurations, or the scores are not comparable.\n- **Truncation.** If your chunks exceed a model's sequence limit, part of every chunk is invisible. This alone can explain a model doing badly.\n- **The same chunks for every model.** Change one variable at a time.\n\n## Afterwards\n\nAdd a hybrid configuration (dense plus BM25 with rank fusion) and a reranker on top of the winner. Both usually help, and now you can say by how much on your own material rather than quoting somebody's blog post.\n\nRe-run the whole grid whenever the corpus changes materially. It is one command by then, and a corpus that has doubled in size or gained a new document type is a different retrieval problem from the one you tuned for."},"check":{"question":"A team compares three embedding models on their own 50-query set. One multilingual model scores far worse than expected. Their chunks are around 800 tokens. What should they check before concluding it is a weak model?","options":[{"text":"Whether the model's index was built with a different similarity metric from the others.","correct":false,"tempting":"Worth standardising, but cosine ranking is unaffected by a consistent metric choice across a single model's index."},{"text":"Whether the model's maximum sequence length is 512 tokens, silently truncating every chunk to its first part.","correct":true},{"text":"Whether the queries were written by someone unfamiliar with the corpus, biasing the labels.","correct":false,"tempting":"A general concern about set quality, but it would depress all three models equally rather than one."},{"text":"Whether the multilingual model needs more training data for their domain.","correct":false,"tempting":"Domain fit is a real factor, but it does not explain a result that a configuration check may fully account for."}]},"prev":{"slug":"comparing-quantisations","title":"Proving what a quantisation cost","url":"https://addaly.com/learn/open-models/comparing-quantisations"},"next":{"slug":"measuring-latency-properly","title":"Measuring latency properly","url":"https://addaly.com/learn/open-models/measuring-latency-properly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Evaluating an embedding model on your corpus', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/evaluating-an-embedding-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Evaluating an embedding model on your corpus","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Retrieval is cheap to evaluate on your own corpus with 50 labelled queries, and the grid of embedding model against chunk size usually shows chunking moving recall more than the model does.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-the-leaderboard-will-not-decide-this","runs":[{"text":"Why the leaderboard will not decide this"}]},{"t":"p","runs":[{"text":"MTEB is the standard embedding benchmark and it has two problems that matter here. Models are routinely trained on the training splits of MTEB’s own tasks, so the aggregate increasingly measures familiarity with those datasets. And the tasks are dominated by English and by web-shaped text; if your corpus is support tickets in Hindi, or contracts, or product listings, the ranking transfers weakly."}]},{"t":"p","runs":[{"text":"The good news is that evaluating retrieval on your own corpus is easy (much easier than evaluating a language model) because there is a right answer and it is cheap to check."}]},{"t":"h2","id":"building-the-set-two-hours","runs":[{"text":"Building the set: two hours"}]},{"t":"ol","start":1,"items":[[{"text":"Take 50 real queries","b":true},{"text":" from your logs. If you have no logs, write them by asking three colleagues what they would type, which is worse but workable."}],[{"text":"For each, find the document or chunk that answers it.","b":true},{"text":" Mark it. Where several are relevant, mark them all. This is the only labour, and 50 queries is about ninety minutes."}],[{"text":"Keep it as JSONL","b":true},{"text":": query, list of relevant chunk identifiers."}]]},{"t":"p","runs":[{"text":"That set is now a permanent asset. It survives model changes, chunking changes and reranker changes, and it is the thing that makes every later decision an hour’s work instead of an argument."}]},{"t":"h2","id":"the-metrics-and-which-to-use","runs":[{"text":"The metrics, and which to use"}]},{"t":"p","runs":[{"text":"Recall at k","b":true},{"text":": the fraction of queries whose relevant document appears in the top k. This is the number for the retrieval stage. Measure it at 100, because that is what the reranker will receive."}]},{"t":"p","runs":[{"text":"MRR","b":true},{"text":": the mean of one divided by the rank of the first relevant result. Sensitive to the top of the list, appropriate when the user sees one answer."}]},{"t":"p","runs":[{"text":"nDCG at 10","b":true},{"text":": discounted gain, handling multiple relevant documents with graded relevance. The standard for a ranked list somebody reads."}]},{"t":"p","runs":[{"text":"Use recall at 100 for the retriever and nDCG at 10 after reranking. Those two numbers separate the two stages, which is the diagnostic that matters, as the reranker lesson showed."}]},{"t":"code","lang":"python","label":"Python","text":"def recall_at_k(results, relevant, k):\n    return sum(bool(set(r[:k]) & set(rel))\n               for r, rel in zip(results, relevant)) / len(relevant)"},{"t":"h2","id":"the-grid-worth-running","runs":[{"text":"The grid worth running"}]},{"t":"p","runs":[{"text":"Chunking usually matters more than the model, and almost nobody tests it. Run the grid:"}]},{"t":"ul","items":[[{"text":"Three embedding models: a small one, a base one, a multilingual one."}],[{"text":"Three chunk sizes: for instance 256, 512 and 1,024 tokens, with a small overlap."}]]},{"t":"p","runs":[{"text":"Nine configurations, each a few minutes to index on 50,000 chunks with a small model on a CPU. The common finding is that the chunk size moves recall more than the model choice does, and that the best chunk size depends on your documents rather than on any general advice."}]},{"t":"p","runs":[{"text":"While you are there, record for each model: "},{"text":"embedding throughput","b":true},{"text":" (documents per second on the hardware you will use), "},{"text":"dimension","b":true},{"text":", and "},{"text":"index size","b":true},{"text":". A model two points better on recall that produces 1024-dimension vectors instead of 384 costs nearly three times the index memory, and that is a real trade rather than a rounding error."}]},{"t":"h2","id":"the-details-that-invalidate-a-comparison","runs":[{"text":"The details that invalidate a comparison"}]},{"t":"ul","items":[[{"text":"Prefixes.","b":true},{"text":" If one model needs "},{"text":"query: ","code":true},{"text":" and "},{"text":"passage: ","code":true},{"text":" and you did not apply them, you have measured the wrong thing. Check the card for every model in the grid."}],[{"text":"Normalisation.","b":true},{"text":" Consistent across all configurations, or the scores are not comparable."}],[{"text":"Truncation.","b":true},{"text":" If your chunks exceed a model’s sequence limit, part of every chunk is invisible. This alone can explain a model doing badly."}],[{"text":"The same chunks for every model.","b":true},{"text":" Change one variable at a time."}]]},{"t":"h2","id":"afterwards","runs":[{"text":"Afterwards"}]},{"t":"p","runs":[{"text":"Add a hybrid configuration (dense plus BM25 with rank fusion) and a reranker on top of the winner. Both usually help, and now you can say by how much on your own material rather than quoting somebody’s blog post."}]},{"t":"p","runs":[{"text":"Re-run the whole grid whenever the corpus changes materially. It is one command by then, and a corpus that has doubled in size or gained a new document type is a different retrieval problem from the one you tuned for."}]}],"check":{"question":[{"text":"A team compares three embedding models on their own 50-query set. One multilingual model scores far worse than expected. Their chunks are around 800 tokens. What should they check before concluding it is a weak model?"}],"options":[{"text":[{"text":"Whether the model’s index was built with a different similarity metric from the others."}],"correct":false,"tempting":[{"text":"Worth standardising, but cosine ranking is unaffected by a consistent metric choice across a single model’s index."}]},{"text":[{"text":"Whether the model’s maximum sequence length is 512 tokens, silently truncating every chunk to its first part."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the queries were written by someone unfamiliar with the corpus, biasing the labels."}],"correct":false,"tempting":[{"text":"A general concern about set quality, but it would depress all three models equally rather than one."}]},{"text":[{"text":"Whether the multilingual model needs more training data for their domain."}],"correct":false,"tempting":[{"text":"Domain fit is a real factor, but it does not explain a result that a configuration check may fully account for."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"comparing-quantisations","title":"Proving what a quantisation cost","minutes":9},"next":{"slug":"measuring-latency-properly","title":"Measuring latency properly","minutes":8}}}