{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"reranking-with-cross-encoders","title":"The second pass that fixes the first","url":"https://addaly.com/learn/hugging-face/reranking-with-cross-encoders","position":54,"of":75,"minutes":8,"answer":"A bi-encoder compares precomputed vectors and scales, while a cross-encoder reads query and document together and judges whether one answers the other.","keyIdea":"A bi-encoder compares precomputed vectors and scales, while a cross-encoder reads query and document together and judges whether one answers the other, so retrieving 50 cheaply and reranking them expensively buys most of the accuracy at a fraction of the cost."},"body":{"format":"markdown","text":"## Two architectures, and why both exist\n\nA **bi-encoder** (everything so far) encodes the query and the document separately and compares vectors. The document's vector is computed once, in advance, so searching a million documents is a million dot products. Fast, and the comparison is crude, because the model never saw the query and the document together.\n\nA **cross-encoder** takes the query and the document as one input and outputs a single relevance score:\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nce = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\nprint(ce.predict([(\"where is my parcel\", \"Your order shipped on Tuesday.\"),\n                  (\"where is my parcel\", \"How to change your password.\")]))\n# [ 8.21 -6.94 ]\n```\n\nEvery query token can attend to every document token, so the model can tell that a document *answers* a question rather than merely being about the same topic. That is exactly the weakness the first lesson named.\n\nThe cost: nothing can be precomputed. Scoring one query against a million documents means a million forward passes. It is completely impractical as a search method.\n\n> Figure: Where the time goes in one retrieval request. Bars (ms): Embed the query 5; Search a million vectors 10; Rerank the top fifty on a CPU 100. About a hundred and twenty milliseconds before any generation begins, and the rerank is four fifths of it. So that is the step to tune, by retrieving fewer candidates or moving the cross-encoder to a GPU. It is also where most of the accuracy comes from, which is why it is worth the time.\n\n## The pattern: retrieve then rerank\n\n```python\ncandidates = search(query, k=50)                      # bi-encoder, milliseconds\npairs = [(query, c[\"text\"]) for c in candidates]\nscores = ce.predict(pairs)                            # 50 forward passes\nbest = [candidates[i] for i in scores.argsort()[::-1][:5]]\n```\n\nFifty forward passes on a small cross-encoder is roughly 50 to 200 ms on a CPU and a few milliseconds on a GPU. You get most of the cross-encoder's accuracy at a tiny fraction of its cost, because the expensive model only ever sees fifty documents.\n\nThe gains are large and well replicated: on standard retrieval benchmarks, reranking the top 50 commonly improves precision at 5 by ten to twenty points over the bi-encoder alone. Very few single changes in this area do that.\n\n## Choosing how many to retrieve\n\nThe reranker can only reorder what it is given. Retrieve 10 and the right answer at rank 40 is lost forever; retrieve 200 and latency grows linearly.\n\nThe number to measure is your bi-encoder's **recall@k**: how often the correct document appears in the top k at all. Retrieve at the k where recall plateaus, typically 30 to 100. If recall@50 is 0.92 and recall@200 is 0.93, retrieving 200 buys you almost nothing and quadruples the rerank cost.\n\nThis is why the thirty-question set exists. Without it, both numbers are guesses.\n\n## Which reranker\n\n- `cross-encoder/ms-marco-MiniLM-L-6-v2`: 90 MB, English, fast, the sensible default.\n- `BAAI/bge-reranker-base` and `-large`: stronger, multilingual coverage including Chinese and several Indic languages.\n- `jina-reranker` family: long inputs.\n\nAll run on CPU. A reranker is usually smaller than the generative model it feeds, and it is the cheapest quality improvement in a retrieval pipeline.\n\n## The language-model reranker, and its trade\n\nYou can also ask a general instruction model to rate relevance from 0 to 10. It works, needs no extra model if you already have one, and is slower, more expensive and less consistent than a trained cross-encoder: the same prompt on the same pair can score differently across runs unless you fix the sampling. Use it when adding a dependency is impossible; prefer the trained model otherwise.\n\n## What reranking cannot fix\n\n- **A document that was never retrieved.** If chunking lost the fact, or the embedding model does not handle your language, no reranker helps. Fix retrieval first.\n- **A document that does not exist.** The corpus is the ceiling.\n- **Diversity.** A reranker will happily put five near-identical chunks at the top. Apply maximal marginal relevance after reranking, not before.\n\n## Latency\n\nA full pipeline (embed query, search, rerank 50) is roughly 5 ms to embed, 10 ms to search, 100 ms to rerank on CPU. Around 120 ms, before any generation. That is acceptable behind a web request and noticeable in a tight loop, and the rerank step dominates, so it is where you tune.\n\n## Do this now\n\nRun your thirty questions with and without reranking and compare recall@1. This is normally the largest single improvement available in a retrieval system, and it takes twenty lines."},"check":{"question":"Why can a cross-encoder distinguish \"How do I cancel?\" from \"How do I renew?\" when a bi-encoder cannot?","options":[{"text":"It is trained on relevance labels rather than similarity pairs, which teaches it to weight decisive words more heavily.","correct":false,"tempting":"Training data differs, but the architectural ability to compare the two texts jointly is what enables the distinction."},{"text":"It processes query and document together, so attention can compare the decisive words directly.","correct":true},{"text":"It outputs an unbounded score rather than a bounded similarity, so small differences are not compressed near the top.","correct":false,"tempting":"Score range affects readability of the output, not the model's ability to tell the pair apart."},{"text":"It encodes documents at query time, so the document vector can be conditioned on the query's length and language.","correct":false,"tempting":"Correctly notes there is no precomputation, but attributes the benefit to conditioning on surface features rather than joint attention."}]},"prev":{"slug":"a-search-index-without-a-database","title":"NumPy first, FAISS second, a database rarely","url":"https://addaly.com/learn/hugging-face/a-search-index-without-a-database"},"next":{"slug":"embedding-images-and-audio","title":"One space for pictures and words","url":"https://addaly.com/learn/hugging-face/embedding-images-and-audio"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The second pass that fixes the first', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/reranking-with-cross-encoders (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The second pass that fixes the first","courseTitle":"Hugging Face, End to End","module":{"index":6,"slug":"embeddings-and-retrieval","title":"Embeddings, and search that understands"},"answer":null,"keyIdea":"A bi-encoder compares precomputed vectors and scales, while a cross-encoder reads query and document together and judges whether one answers the other, so retrieving 50 cheaply and reranking them expensively buys most of the accuracy at a fraction of the cost.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"ms","title":"Where the time goes in one retrieval request","series":[{"label":"Embed the query","value":5},{"label":"Search a million vectors","value":10},{"label":"Rerank the top fifty on a CPU","value":100,"highlight":true}],"caption":"About a hundred and twenty milliseconds before any generation begins, and the rerank is four fifths of it. So that is the step to tune, by retrieving fewer candidates or moving the cross-encoder to a GPU. It is also where most of the accuracy comes from, which is why it is worth the time."},"alt":"Figure: Where the time goes in one retrieval request. Bars (ms): Embed the query 5; Search a million vectors 10; Rerank the top fifty on a CPU 100. About a hundred and twenty milliseconds before any generation begins, and the rerank is four fifths of it. So that is the step to tune, by retrieving fewer candidates or moving the cross-encoder to a GPU. It is also where most of the accuracy comes from, which is why it is worth the time."}},"blocks":[{"t":"h2","id":"two-architectures-and-why-both-exist","runs":[{"text":"Two architectures, and why both exist"}]},{"t":"p","runs":[{"text":"A "},{"text":"bi-encoder","b":true},{"text":" (everything so far) encodes the query and the document separately and compares vectors. The document’s vector is computed once, in advance, so searching a million documents is a million dot products. Fast, and the comparison is crude, because the model never saw the query and the document together."}]},{"t":"p","runs":[{"text":"A "},{"text":"cross-encoder","b":true},{"text":" takes the query and the document as one input and outputs a single relevance score:"}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import CrossEncoder\n\nce = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\nprint(ce.predict([(\"where is my parcel\", \"Your order shipped on Tuesday.\"),\n                  (\"where is my parcel\", \"How to change your password.\")]))\n# [ 8.21 -6.94 ]"},{"t":"p","runs":[{"text":"Every query token can attend to every document token, so the model can tell that a document "},{"text":"answers","i":true},{"text":" a question rather than merely being about the same topic. That is exactly the weakness the first lesson named."}]},{"t":"p","runs":[{"text":"The cost: nothing can be precomputed. Scoring one query against a million documents means a million forward passes. It is completely impractical as a search method."}]},{"t":"h2","id":"the-pattern-retrieve-then-rerank","runs":[{"text":"The pattern: retrieve then rerank"}]},{"t":"code","lang":"python","label":"Python","text":"candidates = search(query, k=50)                      # bi-encoder, milliseconds\npairs = [(query, c[\"text\"]) for c in candidates]\nscores = ce.predict(pairs)                            # 50 forward passes\nbest = [candidates[i] for i in scores.argsort()[::-1][:5]]"},{"t":"p","runs":[{"text":"Fifty forward passes on a small cross-encoder is roughly 50 to 200 ms on a CPU and a few milliseconds on a GPU. You get most of the cross-encoder’s accuracy at a tiny fraction of its cost, because the expensive model only ever sees fifty documents."}]},{"t":"p","runs":[{"text":"The gains are large and well replicated: on standard retrieval benchmarks, reranking the top 50 commonly improves precision at 5 by ten to twenty points over the bi-encoder alone. Very few single changes in this area do that."}]},{"t":"h2","id":"choosing-how-many-to-retrieve","runs":[{"text":"Choosing how many to retrieve"}]},{"t":"p","runs":[{"text":"The reranker can only reorder what it is given. Retrieve 10 and the right answer at rank 40 is lost forever; retrieve 200 and latency grows linearly."}]},{"t":"p","runs":[{"text":"The number to measure is your bi-encoder’s "},{"text":"recall@k","b":true},{"text":": how often the correct document appears in the top k at all. Retrieve at the k where recall plateaus, typically 30 to 100. If recall@50 is 0.92 and recall@200 is 0.93, retrieving 200 buys you almost nothing and quadruples the rerank cost."}]},{"t":"p","runs":[{"text":"This is why the thirty-question set exists. Without it, both numbers are guesses."}]},{"t":"h2","id":"which-reranker","runs":[{"text":"Which reranker"}]},{"t":"ul","items":[[{"text":"cross-encoder/ms-marco-MiniLM-L-6-v2","code":true},{"text":": 90 MB, English, fast, the sensible default."}],[{"text":"BAAI/bge-reranker-base","code":true},{"text":" and "},{"text":"-large","code":true},{"text":": stronger, multilingual coverage including Chinese and several Indic languages."}],[{"text":"jina-reranker","code":true},{"text":" family: long inputs."}]]},{"t":"p","runs":[{"text":"All run on CPU. A reranker is usually smaller than the generative model it feeds, and it is the cheapest quality improvement in a retrieval pipeline."}]},{"t":"h2","id":"the-language-model-reranker-and-its-trade","runs":[{"text":"The language-model reranker, and its trade"}]},{"t":"p","runs":[{"text":"You can also ask a general instruction model to rate relevance from 0 to 10. It works, needs no extra model if you already have one, and is slower, more expensive and less consistent than a trained cross-encoder: the same prompt on the same pair can score differently across runs unless you fix the sampling. Use it when adding a dependency is impossible; prefer the trained model otherwise."}]},{"t":"h2","id":"what-reranking-cannot-fix","runs":[{"text":"What reranking cannot fix"}]},{"t":"ul","items":[[{"text":"A document that was never retrieved.","b":true},{"text":" If chunking lost the fact, or the embedding model does not handle your language, no reranker helps. Fix retrieval first."}],[{"text":"A document that does not exist.","b":true},{"text":" The corpus is the ceiling."}],[{"text":"Diversity.","b":true},{"text":" A reranker will happily put five near-identical chunks at the top. Apply maximal marginal relevance after reranking, not before."}]]},{"t":"h2","id":"latency","runs":[{"text":"Latency"}]},{"t":"p","runs":[{"text":"A full pipeline (embed query, search, rerank 50) is roughly 5 ms to embed, 10 ms to search, 100 ms to rerank on CPU. Around 120 ms, before any generation. That is acceptable behind a web request and noticeable in a tight loop, and the rerank step dominates, so it is where you tune."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Run your thirty questions with and without reranking and compare recall@1. This is normally the largest single improvement available in a retrieval system, and it takes twenty lines."}]}],"check":{"question":[{"text":"Why can a cross-encoder distinguish “How do I cancel?” from “How do I renew?” when a bi-encoder cannot?"}],"options":[{"text":[{"text":"It is trained on relevance labels rather than similarity pairs, which teaches it to weight decisive words more heavily."}],"correct":false,"tempting":[{"text":"Training data differs, but the architectural ability to compare the two texts jointly is what enables the distinction."}]},{"text":[{"text":"It processes query and document together, so attention can compare the decisive words directly."}],"correct":true,"tempting":null},{"text":[{"text":"It outputs an unbounded score rather than a bounded similarity, so small differences are not compressed near the top."}],"correct":false,"tempting":[{"text":"Score range affects readability of the output, not the model’s ability to tell the pair apart."}]},{"text":[{"text":"It encodes documents at query time, so the document vector can be conditioned on the query’s length and language."}],"correct":false,"tempting":[{"text":"Correctly notes there is no precomputation, but attributes the benefit to conditioning on surface features rather than joint attention."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"a-search-index-without-a-database","title":"NumPy first, FAISS second, a database rarely","minutes":9},"next":{"slug":"embedding-images-and-audio","title":"One space for pictures and words","minutes":8}}}