{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"rerankers-and-the-retrieval-stack","title":"Rerankers, and why retrieval is two stages","url":"https://addaly.com/learn/open-models/rerankers-and-the-retrieval-stack","position":16,"of":71,"minutes":9,"answer":"A bi-encoder must compress each document before seeing the query, which is why retrieval is fast and imprecise; a cross-encoder reranker reads query and document together.","keyIdea":"A bi-encoder must compress each document before seeing the query, which is why retrieval is fast and imprecise; a cross-encoder reranker reads query and document together, which is why it is accurate and can only reorder what retrieval already found."},"body":{"format":"markdown","text":"## The limitation that creates the second stage\n\nAn embedding model has to compress a document into a fixed vector *before it knows what anybody will ask*. That is what makes search fast: you embed a million documents once, and each query is a nearest-neighbour lookup. It is also what limits quality, because the compression happens without the query.\n\nA reranker removes that limitation by paying for it. It takes the query and one document *together*, runs them through a transformer, and outputs a single relevance score. Every token of the query can attend to every token of the document. It is far more accurate and it is far too slow to run over a million documents.\n\nSo the standard shape is two stages:\n\n> Figure: Retrieval is two stages, and they fail differently. Steps: 1. The query arrives (One sentence, and the index knows nothing about it yet); 2. Retrieve 100 candidates (Dense vectors plus BM25, fused by rank. A few milliseconds over a million documents); 3. Rerank the 100 (A cross-encoder reads query and document together. Tens to hundreds of milliseconds); 4. Keep the top 5 (This is what separates 'the payment succeeded' from 'the payment failed'); 5. Into the model's context (Five paragraphs beats a hundred thousand tokens on cost, speed, accuracy and citation). A reranker cannot find what the first stage missed; it only reorders what it is given. So measure recall at 100 for the retriever first, then precision in the top five after reranking. Teams reporting one end-to-end number spend weeks tuning the wrong stage.\n\n1. **Retrieve** the top 50 to 100 candidates with the embedding model, in a few milliseconds.\n2. **Rerank** those candidates with the cross-encoder, in tens to hundreds of milliseconds, and keep the top 5 to 10.\n\nThe vocabulary: the embedding model is a **bi-encoder** because it encodes each side separately; the reranker is a **cross-encoder** because it encodes them jointly.\n\n## What it is worth\n\nOn typical document collections, adding a reranker moves the quality of the top 5 results more than upgrading from a small embedding model to a large one, and it costs less to change. In a retrieval-augmented system, the top 5 is what the language model sees, so this is usually the highest-return single change available.\n\nThe mechanism explains where it helps most: cases where two candidates are topically identical but only one answers the question. The pair *the payment succeeded* and *the payment failed* from the previous lesson is exactly the case a bi-encoder cannot separate and a cross-encoder can, because the cross-encoder sees the word *failed* in the presence of the word the user actually asked about.\n\n## The open rerankers\n\n- **bge-reranker-v2-m3.** Multilingual, permissive, the reliable default.\n- **Qwen3-Reranker.** Several sizes, recent, strong.\n- **mxbai-rerank** and **jina-reranker** are alternatives worth measuring against.\n- **ColBERT** and its successors take a middle path called late interaction: they keep one vector per token rather than one per document, and compare token to token. More accurate than a bi-encoder, faster than a cross-encoder, and much larger indexes. Worth knowing exists; worth using when the index size is affordable.\n\n```python\nfrom sentence_transformers import CrossEncoder\nce = CrossEncoder(\"BAAI/bge-reranker-v2-m3\", max_length=512)\nscores = ce.predict([(query, doc) for doc in candidates])\ntop = [d for _, d in sorted(zip(scores, candidates), reverse=True)][:5]\n```\n\nA base-sized reranker scores roughly 50 pairs in a few hundred milliseconds on a CPU and far faster on a GPU. Budget for it: reranking 100 candidates costs more than retrieving them by a factor of ten or more.\n\n## The other half of the first stage\n\nDense retrieval has a specific blind spot: exact strings. Product codes, error numbers, surnames, an unusual acronym. The embedding compresses these into a general neighbourhood, and the exact match is not recoverable.\n\n**BM25** (classic keyword scoring, available in every search engine and in small libraries) has the opposite profile. It finds exact tokens and misses paraphrases.\n\nRunning both and combining them is called hybrid retrieval, and the usual combination method is reciprocal rank fusion, which merges two ranked lists by rank rather than by score, so you do not have to make two incomparable scores comparable:\n\n```\nscore(d) = sum over lists of  1 / (60 + rank_of_d_in_list)\n```\n\nHybrid plus a reranker is the configuration most production systems converge on, and it is not fashion; it is two different failure modes being covered by two different mechanisms.\n\n## The limit worth stating\n\n**A reranker cannot find what the first stage missed.** It only reorders what it is given. If the correct document is not in the top 100 retrieved, no reranker will produce it, and every symptom will look like a reranker problem.\n\nSo measure the two stages separately. Recall at 100 for the retriever tells you whether the answer is even in the candidate set; that is the number to fix first. Then measure precision in the top 5 after reranking. Teams that report a single end-to-end number spend weeks tuning the wrong stage."},"check":{"question":"A support search system returns the wrong article about 30% of the time. Recall at 100 for the retriever is 96%, and precision in the top 5 after reranking is 70%. Where should the work go?","options":[{"text":"Into the retriever, since 96% still means one query in 25 fails outright.","correct":false,"tempting":"Worth fixing eventually, but 4% missing candidates cannot explain a 30% top-5 failure rate."},{"text":"Into a larger embedding model, which would improve both stages at once.","correct":false,"tempting":"A better embedding model mostly improves candidate recall, which is already the healthy number here."},{"text":"Into the reranker or the ranking signals it uses, because the right document is nearly always in the candidate set and is being ordered below others.","correct":true},{"text":"Into chunking, since the top-5 failures indicate documents are being split badly.","correct":false,"tempting":"Bad chunking usually shows up as poor recall at 100, because a split answer is hard to retrieve at all."}]},"prev":{"slug":"open-embedding-models","title":"Embeddings you can host yourself","url":"https://addaly.com/learn/open-models/open-embedding-models"},"next":{"slug":"open-speech-models","title":"Speech, in both directions","url":"https://addaly.com/learn/open-models/open-speech-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rerankers, and why retrieval is two stages', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/rerankers-and-the-retrieval-stack (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rerankers, and why retrieval is two stages","courseTitle":"The Open Model Ecosystem","module":{"index":2,"slug":"the-map-of-the-ecosystem","title":"The map, and how to read it"},"answer":null,"keyIdea":"A bi-encoder must compress each document before seeing the query, which is why retrieval is fast and imprecise; a cross-encoder reranker reads query and document together, which is why it is accurate and can only reorder what retrieval already found.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"The query arrives","detail":"One sentence, and the index knows nothing about it yet"},{"label":"Retrieve 100 candidates","detail":"Dense vectors plus BM25, fused by rank. A few milliseconds over a million documents"},{"label":"Rerank the 100","detail":"A cross-encoder reads query and document together. Tens to hundreds of milliseconds"},{"label":"Keep the top 5","detail":"This is what separates ‘the payment succeeded’ from ‘the payment failed’"},{"label":"Into the model’s context","detail":"Five paragraphs beats a hundred thousand tokens on cost, speed, accuracy and citation"}],"title":"Retrieval is two stages, and they fail differently","caption":"A reranker cannot find what the first stage missed; it only reorders what it is given. So measure recall at 100 for the retriever first, then precision in the top five after reranking. Teams reporting one end-to-end number spend weeks tuning the wrong stage."},"alt":"Figure: Retrieval is two stages, and they fail differently. Steps: 1. The query arrives (One sentence, and the index knows nothing about it yet); 2. Retrieve 100 candidates (Dense vectors plus BM25, fused by rank. A few milliseconds over a million documents); 3. Rerank the 100 (A cross-encoder reads query and document together. Tens to hundreds of milliseconds); 4. Keep the top 5 (This is what separates ‘the payment succeeded’ from ‘the payment failed’); 5. Into the model’s context (Five paragraphs beats a hundred thousand tokens on cost, speed, accuracy and citation). A reranker cannot find what the first stage missed; it only reorders what it is given. So measure recall at 100 for the retriever first, then precision in the top five after reranking. Teams reporting one end-to-end number spend weeks tuning the wrong stage."}},"blocks":[{"t":"h2","id":"the-limitation-that-creates-the-second-stage","runs":[{"text":"The limitation that creates the second stage"}],"role":"limit"},{"t":"p","runs":[{"text":"An embedding model has to compress a document into a fixed vector "},{"text":"before it knows what anybody will ask","i":true},{"text":". That is what makes search fast: you embed a million documents once, and each query is a nearest-neighbour lookup. It is also what limits quality, because the compression happens without the query."}]},{"t":"p","runs":[{"text":"A reranker removes that limitation by paying for it. It takes the query and one document "},{"text":"together","i":true},{"text":", runs them through a transformer, and outputs a single relevance score. Every token of the query can attend to every token of the document. It is far more accurate and it is far too slow to run over a million documents."}]},{"t":"p","runs":[{"text":"So the standard shape is two stages:"}]},{"t":"ol","start":1,"items":[[{"text":"Retrieve","b":true},{"text":" the top 50 to 100 candidates with the embedding model, in a few milliseconds."}],[{"text":"Rerank","b":true},{"text":" those candidates with the cross-encoder, in tens to hundreds of milliseconds, and keep the top 5 to 10."}]]},{"t":"p","runs":[{"text":"The vocabulary: the embedding model is a "},{"text":"bi-encoder","b":true},{"text":" because it encodes each side separately; the reranker is a "},{"text":"cross-encoder","b":true},{"text":" because it encodes them jointly."}]},{"t":"h2","id":"what-it-is-worth","runs":[{"text":"What it is worth"}]},{"t":"p","runs":[{"text":"On typical document collections, adding a reranker moves the quality of the top 5 results more than upgrading from a small embedding model to a large one, and it costs less to change. In a retrieval-augmented system, the top 5 is what the language model sees, so this is usually the highest-return single change available."}]},{"t":"p","runs":[{"text":"The mechanism explains where it helps most: cases where two candidates are topically identical but only one answers the question. The pair "},{"text":"the payment succeeded","i":true},{"text":" and "},{"text":"the payment failed","i":true},{"text":" from the previous lesson is exactly the case a bi-encoder cannot separate and a cross-encoder can, because the cross-encoder sees the word "},{"text":"failed","i":true},{"text":" in the presence of the word the user actually asked about."}]},{"t":"h2","id":"the-open-rerankers","runs":[{"text":"The open rerankers"}]},{"t":"ul","items":[[{"text":"bge-reranker-v2-m3.","b":true},{"text":" Multilingual, permissive, the reliable default."}],[{"text":"Qwen3-Reranker.","b":true},{"text":" Several sizes, recent, strong."}],[{"text":"mxbai-rerank","b":true},{"text":" and "},{"text":"jina-reranker","b":true},{"text":" are alternatives worth measuring against."}],[{"text":"ColBERT","b":true},{"text":" and its successors take a middle path called late interaction: they keep one vector per token rather than one per document, and compare token to token. More accurate than a bi-encoder, faster than a cross-encoder, and much larger indexes. Worth knowing exists; worth using when the index size is affordable."}]]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import CrossEncoder\nce = CrossEncoder(\"BAAI/bge-reranker-v2-m3\", max_length=512)\nscores = ce.predict([(query, doc) for doc in candidates])\ntop = [d for _, d in sorted(zip(scores, candidates), reverse=True)][:5]"},{"t":"p","runs":[{"text":"A base-sized reranker scores roughly 50 pairs in a few hundred milliseconds on a CPU and far faster on a GPU. Budget for it: reranking 100 candidates costs more than retrieving them by a factor of ten or more."}]},{"t":"h2","id":"the-other-half-of-the-first-stage","runs":[{"text":"The other half of the first stage"}]},{"t":"p","runs":[{"text":"Dense retrieval has a specific blind spot: exact strings. Product codes, error numbers, surnames, an unusual acronym. The embedding compresses these into a general neighbourhood, and the exact match is not recoverable."}]},{"t":"p","runs":[{"text":"BM25","b":true},{"text":" (classic keyword scoring, available in every search engine and in small libraries) has the opposite profile. It finds exact tokens and misses paraphrases."}]},{"t":"p","runs":[{"text":"Running both and combining them is called hybrid retrieval, and the usual combination method is reciprocal rank fusion, which merges two ranked lists by rank rather than by score, so you do not have to make two incomparable scores comparable:"}]},{"t":"code","lang":"","label":"Text","text":"score(d) = sum over lists of  1 / (60 + rank_of_d_in_list)"},{"t":"p","runs":[{"text":"Hybrid plus a reranker is the configuration most production systems converge on, and it is not fashion; it is two different failure modes being covered by two different mechanisms."}]},{"t":"h2","id":"the-limit-worth-stating","runs":[{"text":"The limit worth stating"}],"role":"limit"},{"t":"p","runs":[{"text":"A reranker cannot find what the first stage missed.","b":true},{"text":" It only reorders what it is given. If the correct document is not in the top 100 retrieved, no reranker will produce it, and every symptom will look like a reranker problem."}]},{"t":"p","runs":[{"text":"So measure the two stages separately. Recall at 100 for the retriever tells you whether the answer is even in the candidate set; that is the number to fix first. Then measure precision in the top 5 after reranking. Teams that report a single end-to-end number spend weeks tuning the wrong stage."}]}],"check":{"question":[{"text":"A support search system returns the wrong article about 30% of the time. Recall at 100 for the retriever is 96%, and precision in the top 5 after reranking is 70%. Where should the work go?"}],"options":[{"text":[{"text":"Into the retriever, since 96% still means one query in 25 fails outright."}],"correct":false,"tempting":[{"text":"Worth fixing eventually, but 4% missing candidates cannot explain a 30% top-5 failure rate."}]},{"text":[{"text":"Into a larger embedding model, which would improve both stages at once."}],"correct":false,"tempting":[{"text":"A better embedding model mostly improves candidate recall, which is already the healthy number here."}]},{"text":[{"text":"Into the reranker or the ranking signals it uses, because the right document is nearly always in the candidate set and is being ordered below others."}],"correct":true,"tempting":null},{"text":[{"text":"Into chunking, since the top-5 failures indicate documents are being split badly."}],"correct":false,"tempting":[{"text":"Bad chunking usually shows up as poor recall at 100, because a split answer is hard to retrieve at all."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"open-embedding-models","title":"Embeddings you can host yourself","minutes":10},"next":{"slug":"open-speech-models","title":"Speech, in both directions","minutes":9}}}