{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"training-a-reranker","title":"Training a reranker","url":"https://addaly.com/learn/fine-tuning/training-a-reranker","position":59,"of":84,"minutes":8,"answer":"A cross-encoder reads query and passage together so it ranks far better than a bi-encoder, but it must run once per candidate, which is why it reranks 50 results rather than searching a corpus.","keyIdea":"A cross-encoder reads query and passage together so it ranks far better than a bi-encoder, but it must run once per candidate, which is why it reranks 50 results rather than searching a corpus, and why the pretrained one should be measured before you train your own."},"body":{"format":"markdown","text":"## Why a cross-encoder is better\n\nAn embedding model encodes the query and the passage separately, into two vectors that are compared afterwards. Whatever the passage says about the query has to survive being compressed into a fixed vector before the query is ever seen.\n\nA cross-encoder puts both into the model at once (`[query] [SEP] [passage]`) and produces one relevance score. Every token of the query can attend to every token of the passage. That is why rerankers beat bi-encoders consistently, often by a wide margin on the harder queries.\n\nThe price is that the score cannot be precomputed. A bi-encoder embeds your corpus once, and a query is one vector comparison against an index. A cross-encoder must run the model once per candidate, at query time.\n\n## The two-stage architecture\n\nSo you use both:\n\n1. **Retrieve 50 to 100 candidates** with the bi-encoder plus BM25. Fast, over the whole corpus.\n2. **Rerank those candidates** with the cross-encoder. Slow per item, but only 50 items.\n3. **Pass the top 5** to the generator.\n\nThe arithmetic that makes this work: a 22M-parameter cross-encoder scores 50 query-passage pairs in roughly 15 to 40 milliseconds on a GPU, or a few hundred milliseconds on a CPU. Doing the same over a 200,000-document corpus would take minutes. The two-stage design buys cross-encoder quality at bi-encoder latency.\n\n## Training it\n\n```python\nfrom sentence_transformers import CrossEncoder\nfrom sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss\n\nmodel = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L6-v2\", num_labels=1)\n```\n\nThe data is triples of query, passage and label. With binary cross-entropy the label is 1 for relevant and 0 for not, and the ratio matters. Around four negatives per positive is a common starting point; far more negatives and the model learns to say \"no\" to everything, which scores well on accuracy and ranks nothing.\n\nThe negatives should be **mined from your retriever's actual output**, as with embedding training. A reranker's entire job is to sort the candidates your first stage returns, so training it on random negatives trains it for a task it will never face.\n\n## The listwise alternative\n\nBinary labels treat each pair independently, but ranking is about order. Losses that compare a positive against several negatives for the same query within one forward group (the family that recent reranker work uses) optimise the ordering directly and generally produce better rankings from the same data. `CachedMultipleNegativesRankingLoss` has a cross-encoder counterpart in recent sentence-transformers versions, and it is worth preferring where available.\n\n## Evaluating a reranker properly\n\nThe metric is the ranking quality of the *reranked list*, not classification accuracy on pairs. NDCG at 10 and MRR at 10 are the standard measures.\n\nThe honest comparison is against your first stage, on the same candidates:\n\n```\nstage 1 (bi-encoder + BM25), NDCG@10 = 0.61\n+ off-the-shelf reranker,    NDCG@10 = 0.74\n+ fine-tuned reranker,       NDCG@10 = 0.79\n```\n\nNote what that shows. The off-the-shelf reranker delivered most of the gain, and fine-tuning added a smaller increment. **Try the pretrained reranker before training one.** The public cross-encoders are trained on very large relevance datasets and are strong out of the box; on many corpora they are good enough, and you have spent nothing.\n\n## The latency budget\n\nThe decision that constrains everything is how many candidates you can afford to rerank.\n\n- 22M parameters, 50 candidates: tens of milliseconds on a GPU. Comfortable.\n- 278M parameters, 50 candidates: low hundreds of milliseconds. Noticeable in an interactive product.\n- A large LLM-based reranker: hundreds of milliseconds to seconds. Only for offline or high-value queries.\n\nHalving the candidate count halves the cost, and the recall you lose by reranking 25 instead of 50 is usually small: measure it, because it is the cheapest latency win in the pipeline.\n\n## Where it belongs in the course\n\nBack in module one the argument was that retrieval should be fixed before the generator. This lesson and the previous one are what that means concretely. A fine-tuned embedding model raises what can be found; a reranker raises what reaches the top. Together they typically cost a day and a few hundred megabytes, and they move end-to-end quality further than a generator fine-tune costing ten times as much.\n\n> Figure: Reranking, measured on the same candidates. Bars (NDCG@10): Bi-encoder plus BM25 alone 0.61; Plus an off-the-shelf reranker 0.74; Plus a reranker fine-tuned on your data 0.79. The off-the-shelf cross-encoder delivered most of the gain and cost nothing. Measure the pretrained one before you train your own, and mine your negatives from your retriever's real output rather than at random."},"check":{"question":"Why can a cross-encoder not replace the bi-encoder for first-stage retrieval over a 200,000-document corpus?","options":[{"text":"Its score depends on the query, so nothing can be precomputed and the model must run once per document at query time.","correct":true},{"text":"Cross-encoders produce scores rather than vectors, and ranking requires a vector space to compute distances in.","correct":false,"tempting":"Treats a score as insufficient for ranking, when a score per candidate is exactly what ranking needs."},{"text":"Cross-encoders have shorter context windows, so most corpus documents would be truncated beyond usefulness.","correct":false,"tempting":"Notes a real constraint on long passages that applies equally when reranking, and is not why first-stage retrieval is infeasible."},{"text":"Cross-encoders are trained on relevance labels rather than similarity, so their scores are not comparable across different queries.","correct":false,"tempting":"Raises a calibration point that is true and irrelevant, since ranking within a single query needs no cross-query comparability."}]},"prev":{"slug":"fine-tuning-an-embedding-model","title":"Fine-tuning an embedding model","url":"https://addaly.com/learn/fine-tuning/fine-tuning-an-embedding-model"},"next":{"slug":"fine-tuning-a-classifier","title":"Fine-tuning a classifier properly","url":"https://addaly.com/learn/fine-tuning/fine-tuning-a-classifier"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Training a reranker', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/training-a-reranker (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Training a reranker","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"A cross-encoder reads query and passage together so it ranks far better than a bi-encoder, but it must run once per candidate, which is why it reranks 50 results rather than searching a corpus, and why the pretrained one should be measured before you train your own.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"NDCG@10","title":"Reranking, measured on the same candidates","series":[{"label":"Bi-encoder plus BM25 alone","value":0.61},{"label":"Plus an off-the-shelf reranker","value":0.74,"highlight":true},{"label":"Plus a reranker fine-tuned on your data","value":0.79}],"caption":"The off-the-shelf cross-encoder delivered most of the gain and cost nothing. Measure the pretrained one before you train your own, and mine your negatives from your retriever’s real output rather than at random."},"alt":"Figure: Reranking, measured on the same candidates. Bars (NDCG@10): Bi-encoder plus BM25 alone 0.61; Plus an off-the-shelf reranker 0.74; Plus a reranker fine-tuned on your data 0.79. The off-the-shelf cross-encoder delivered most of the gain and cost nothing. Measure the pretrained one before you train your own, and mine your negatives from your retriever’s real output rather than at random."}},"blocks":[{"t":"h2","id":"why-a-cross-encoder-is-better","runs":[{"text":"Why a cross-encoder is better"}]},{"t":"p","runs":[{"text":"An embedding model encodes the query and the passage separately, into two vectors that are compared afterwards. Whatever the passage says about the query has to survive being compressed into a fixed vector before the query is ever seen."}]},{"t":"p","runs":[{"text":"A cross-encoder puts both into the model at once ("},{"text":"[query] [SEP] [passage]","code":true},{"text":") and produces one relevance score. Every token of the query can attend to every token of the passage. That is why rerankers beat bi-encoders consistently, often by a wide margin on the harder queries."}]},{"t":"p","runs":[{"text":"The price is that the score cannot be precomputed. A bi-encoder embeds your corpus once, and a query is one vector comparison against an index. A cross-encoder must run the model once per candidate, at query time."}]},{"t":"h2","id":"the-two-stage-architecture","runs":[{"text":"The two-stage architecture"}]},{"t":"p","runs":[{"text":"So you use both:"}]},{"t":"ol","start":1,"items":[[{"text":"Retrieve 50 to 100 candidates","b":true},{"text":" with the bi-encoder plus BM25. Fast, over the whole corpus."}],[{"text":"Rerank those candidates","b":true},{"text":" with the cross-encoder. Slow per item, but only 50 items."}],[{"text":"Pass the top 5","b":true},{"text":" to the generator."}]]},{"t":"p","runs":[{"text":"The arithmetic that makes this work: a 22M-parameter cross-encoder scores 50 query-passage pairs in roughly 15 to 40 milliseconds on a GPU, or a few hundred milliseconds on a CPU. Doing the same over a 200,000-document corpus would take minutes. The two-stage design buys cross-encoder quality at bi-encoder latency."}]},{"t":"h2","id":"training-it","runs":[{"text":"Training it"}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import CrossEncoder\nfrom sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss\n\nmodel = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L6-v2\", num_labels=1)"},{"t":"p","runs":[{"text":"The data is triples of query, passage and label. With binary cross-entropy the label is 1 for relevant and 0 for not, and the ratio matters. Around four negatives per positive is a common starting point; far more negatives and the model learns to say “no” to everything, which scores well on accuracy and ranks nothing."}]},{"t":"p","runs":[{"text":"The negatives should be "},{"text":"mined from your retriever’s actual output","b":true},{"text":", as with embedding training. A reranker’s entire job is to sort the candidates your first stage returns, so training it on random negatives trains it for a task it will never face."}]},{"t":"h2","id":"the-listwise-alternative","runs":[{"text":"The listwise alternative"}]},{"t":"p","runs":[{"text":"Binary labels treat each pair independently, but ranking is about order. Losses that compare a positive against several negatives for the same query within one forward group (the family that recent reranker work uses) optimise the ordering directly and generally produce better rankings from the same data. "},{"text":"CachedMultipleNegativesRankingLoss","code":true},{"text":" has a cross-encoder counterpart in recent sentence-transformers versions, and it is worth preferring where available."}]},{"t":"h2","id":"evaluating-a-reranker-properly","runs":[{"text":"Evaluating a reranker properly"}]},{"t":"p","runs":[{"text":"The metric is the ranking quality of the "},{"text":"reranked list","i":true},{"text":", not classification accuracy on pairs. NDCG at 10 and MRR at 10 are the standard measures."}]},{"t":"p","runs":[{"text":"The honest comparison is against your first stage, on the same candidates:"}]},{"t":"code","lang":"","label":"Text","text":"stage 1 (bi-encoder + BM25), NDCG@10 = 0.61\n+ off-the-shelf reranker,    NDCG@10 = 0.74\n+ fine-tuned reranker,       NDCG@10 = 0.79"},{"t":"p","runs":[{"text":"Note what that shows. The off-the-shelf reranker delivered most of the gain, and fine-tuning added a smaller increment. "},{"text":"Try the pretrained reranker before training one.","b":true},{"text":" The public cross-encoders are trained on very large relevance datasets and are strong out of the box; on many corpora they are good enough, and you have spent nothing."}]},{"t":"h2","id":"the-latency-budget","runs":[{"text":"The latency budget"}]},{"t":"p","runs":[{"text":"The decision that constrains everything is how many candidates you can afford to rerank."}]},{"t":"ul","items":[[{"text":"22M parameters, 50 candidates: tens of milliseconds on a GPU. Comfortable."}],[{"text":"278M parameters, 50 candidates: low hundreds of milliseconds. Noticeable in an interactive product."}],[{"text":"A large LLM-based reranker: hundreds of milliseconds to seconds. Only for offline or high-value queries."}]]},{"t":"p","runs":[{"text":"Halving the candidate count halves the cost, and the recall you lose by reranking 25 instead of 50 is usually small: measure it, because it is the cheapest latency win in the pipeline."}]},{"t":"h2","id":"where-it-belongs-in-the-course","runs":[{"text":"Where it belongs in the course"}]},{"t":"p","runs":[{"text":"Back in module one the argument was that retrieval should be fixed before the generator. This lesson and the previous one are what that means concretely. A fine-tuned embedding model raises what can be found; a reranker raises what reaches the top. Together they typically cost a day and a few hundred megabytes, and they move end-to-end quality further than a generator fine-tune costing ten times as much."}]}],"check":{"question":[{"text":"Why can a cross-encoder not replace the bi-encoder for first-stage retrieval over a 200,000-document corpus?"}],"options":[{"text":[{"text":"Its score depends on the query, so nothing can be precomputed and the model must run once per document at query time."}],"correct":true,"tempting":null},{"text":[{"text":"Cross-encoders produce scores rather than vectors, and ranking requires a vector space to compute distances in."}],"correct":false,"tempting":[{"text":"Treats a score as insufficient for ranking, when a score per candidate is exactly what ranking needs."}]},{"text":[{"text":"Cross-encoders have shorter context windows, so most corpus documents would be truncated beyond usefulness."}],"correct":false,"tempting":[{"text":"Notes a real constraint on long passages that applies equally when reranking, and is not why first-stage retrieval is infeasible."}]},{"text":[{"text":"Cross-encoders are trained on relevance labels rather than similarity, so their scores are not comparable across different queries."}],"correct":false,"tempting":[{"text":"Raises a calibration point that is true and irrelevant, since ranking within a single query needs no cross-query comparability."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"fine-tuning-an-embedding-model","title":"Fine-tuning an embedding model","minutes":10},"next":{"slug":"fine-tuning-a-classifier","title":"Fine-tuning a classifier properly","minutes":9}}}