{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"reranking-to-fit-the-budget","title":"Reranking, and why it reads better than the retriever","url":"https://addaly.com/learn/context-engineering/reranking-to-fit-the-budget","position":26,"of":87,"minutes":9,"answer":"A cross-encoder scores query and passage together, so it can see negation and specificity a bi-encoder's separate vectors cannot: retrieve broadly, rerank a shortlist.","keyIdea":"A cross-encoder scores query and passage together, so it can see negation and specificity a bi-encoder's separate vectors cannot: retrieve broadly, rerank a shortlist, and check the reranker's input limit before blaming it for long chunks it never fully read."},"body":{"format":"markdown","text":"## Two ways to compare a query with a passage\n\nA bi-encoder (an ordinary embedding model) encodes the query and the passage separately, into two vectors, and compares them. Separately is the key word: the passage's vector was computed months ago with no knowledge of any query. That is what makes it fast enough to search millions of documents, and it is also what limits it. All the interaction between the question and the text has to be squeezed through a single dot product between two summaries.\n\nA cross-encoder puts the query and the passage into the model *together* and outputs one relevance score. Every token of the question can attend to every token of the passage. It can notice that the passage answers a different question about the same subject, that it contains a negation, that the number it gives is for a different plan.\n\nThe cost is that you cannot precompute anything. Scoring 50 passages is 50 forward passes at query time, so a cross-encoder cannot search a corpus; it can only re-order a shortlist.\n\nHence the standard shape: **retrieve 50 cheaply, rerank them, keep 5.**\n\n> Figure: Retrieve broadly, rerank a shortlist, keep five. Steps: 1. Retrieve fifty, cheaply (Vectors and keywords together: the only stage that can miss a passage outright); 2. Score each one together (Query and passage in the model at once, so negation and specificity are visible); 3. Check the input limit first (Often 512 tokens for both, so a 900-token chunk is cut before it is scored); 4. Keep what clears the floor (Two passages for an easy question, eight for a hard one, up to the budget); 5. Abstain if nothing does (Say you do not have it, with no generation call at all). The free path is genuinely competitive here: a small cross-encoder is about 280MB and scores fifty short passages on a laptop CPU in a few hundred milliseconds. The paid services are sometimes better and are worth it when latency at scale matters more than the bill.\n\n## What it is worth\n\nReranking is usually the highest-return single addition to a retrieval pipeline, ahead of a better embedding model and well ahead of prompt tuning. The reason is structural: your retriever's job becomes recall at 50, which is a much easier target than precision at 5, and the reranker supplies the precision.\n\nIt also directly serves the budget. The k-curve lesson showed accuracy peaking and falling as passages are added. A reranker lets you sit at the good end of that curve: five passages that are the best five, rather than ten hedged ones.\n\n## The free path is competitive here\n\n```python\nfrom sentence_transformers import CrossEncoder\nce = CrossEncoder(\"BAAI/bge-reranker-base\")          # ~280MB, CPU is fine\npairs = [(query, c.text) for c in candidates]\nfor c, s in zip(candidates, ce.predict(pairs)):\n    c.rerank_score = float(s)\ncandidates.sort(key=lambda c: c.rerank_score, reverse=True)\n```\n\nOn a laptop CPU, scoring 50 short passages with a small cross-encoder takes on the order of a few hundred milliseconds; on a modest GPU it is tens of milliseconds. `bge-reranker-v2-m3` is the multilingual option and handles Indic and East Asian languages far better than English-only models: worth the extra size if your users do not all write in English.\n\nThe paid services (Cohere Rerank, Voyage, Jina) are strong, sometimes better, and priced per thousand documents scored. They are worth the money when latency at scale matters more than the bill. They are not required to learn or to ship this.\n\n## The truncation trap\n\nThe detail that catches almost everybody: **cross-encoders have a short maximum input**, often 512 tokens for query and passage combined. A 900-token chunk gets silently truncated, and the reranker scores only its first half.\n\nSo a passage whose relevant sentence is near the end scores as though that sentence does not exist, and it drops out of your top five. The symptom is a reranker that seems to make things worse on long chunks, which teams then blame on the reranker rather than on the truncation.\n\nThree fixes. Keep chunks inside the reranker's window at ingestion. Score the chunk in overlapping halves and take the maximum. Or use a long-context reranker and pay for it in latency. Check your model's actual limit rather than assuming; it is one line in the model card and it changes the design.\n\n## Rerank scores are worth keeping\n\nUnlike a cosine similarity, a cross-encoder score is trained on relevance labels, so it means something closer to what you want. Two uses beyond ordering.\n\n**Abstention.** Calibrate a floor on judged examples, and when the top reranked score falls below it, return \"I do not have this\" without a generation call. This is a better abstention signal than the embedding score gap from the earlier lesson, for the obvious reason.\n\n**Cutting adaptively.** Instead of always taking five, take everything above the floor up to the token budget. Easy questions take two passages; hard ones take eight. Do not put the score in the context: the metadata lesson explains why.\n\n## Where reranking does not help\n\nIt cannot retrieve what the first stage missed. If your gold passage is not in the 50, the reranker will never see it, and improving the reranker is wasted effort. Measure recall at 50 first: if it is 70%, your ceiling is 70% however good the reranker is, and the work belongs in the first stage.\n\nIt also adds a serial step to the request path. On a latency-critical surface (typeahead, an in-editor suggestion) a few hundred milliseconds may cost more than the quality is worth. Reranking is a quality-for-latency trade and it should be made deliberately, not because a tutorial included it."},"check":{"question":"A pipeline retrieves 50 candidates with recall@50 of 71%, adds a strong reranker, and end-to-end accuracy barely moves. What does this indicate?","options":[{"text":"The reranker's scores are uncalibrated, so the cut-off is admitting weak passages.","correct":false,"tempting":"Calibration affects where you cut, not whether the correct passage is available to be ranked at all."},{"text":"The first stage misses the correct passage on 29% of queries, and reranking cannot recover what was never retrieved.","correct":true},{"text":"The reranker and the embedding model disagree, so their orderings cancel out.","correct":false,"tempting":"Treats reranking as a vote when it wholly replaces the ordering of the shortlist it is given."},{"text":"Reranking only helps when k is large, and five passages is too few for it to matter.","correct":false,"tempting":"Inverts the relationship: reranking matters most precisely when few passages will be kept."}]},"prev":{"slug":"keywords-and-vectors-together","title":"Keywords still win, so run both","url":"https://addaly.com/learn/context-engineering/keywords-and-vectors-together"},"next":{"slug":"structured-sources-in-context","title":"Rows, records and schemas in the window","url":"https://addaly.com/learn/context-engineering/structured-sources-in-context"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reranking, and why it reads better than the retriever', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/reranking-to-fit-the-budget (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reranking, and why it reads better than the retriever","courseTitle":"Context Engineering","module":{"index":3,"slug":"retrieval-as-a-pipeline","title":"Retrieval as a context pipeline"},"answer":null,"keyIdea":"A cross-encoder scores query and passage together, so it can see negation and specificity a bi-encoder’s separate vectors cannot: retrieve broadly, rerank a shortlist, and check the reranker’s input limit before blaming it for long chunks it never fully read.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Retrieve fifty, cheaply","detail":"Vectors and keywords together: the only stage that can miss a passage outright"},{"label":"Score each one together","detail":"Query and passage in the model at once, so negation and specificity are visible"},{"label":"Check the input limit first","detail":"Often 512 tokens for both, so a 900-token chunk is cut before it is scored"},{"label":"Keep what clears the floor","detail":"Two passages for an easy question, eight for a hard one, up to the budget"},{"label":"Abstain if nothing does","detail":"Say you do not have it, with no generation call at all"}],"title":"Retrieve broadly, rerank a shortlist, keep five","caption":"The free path is genuinely competitive here: a small cross-encoder is about 280MB and scores fifty short passages on a laptop CPU in a few hundred milliseconds. The paid services are sometimes better and are worth it when latency at scale matters more than the bill."},"alt":"Figure: Retrieve broadly, rerank a shortlist, keep five. Steps: 1. Retrieve fifty, cheaply (Vectors and keywords together: the only stage that can miss a passage outright); 2. Score each one together (Query and passage in the model at once, so negation and specificity are visible); 3. Check the input limit first (Often 512 tokens for both, so a 900-token chunk is cut before it is scored); 4. Keep what clears the floor (Two passages for an easy question, eight for a hard one, up to the budget); 5. Abstain if nothing does (Say you do not have it, with no generation call at all). The free path is genuinely competitive here: a small cross-encoder is about 280MB and scores fifty short passages on a laptop CPU in a few hundred milliseconds. The paid services are sometimes better and are worth it when latency at scale matters more than the bill."}},"blocks":[{"t":"h2","id":"two-ways-to-compare-a-query-with-a-passage","runs":[{"text":"Two ways to compare a query with a passage"}]},{"t":"p","runs":[{"text":"A bi-encoder (an ordinary embedding model) encodes the query and the passage separately, into two vectors, and compares them. Separately is the key word: the passage’s vector was computed months ago with no knowledge of any query. That is what makes it fast enough to search millions of documents, and it is also what limits it. All the interaction between the question and the text has to be squeezed through a single dot product between two summaries."}]},{"t":"p","runs":[{"text":"A cross-encoder puts the query and the passage into the model "},{"text":"together","i":true},{"text":" and outputs one relevance score. Every token of the question can attend to every token of the passage. It can notice that the passage answers a different question about the same subject, that it contains a negation, that the number it gives is for a different plan."}]},{"t":"p","runs":[{"text":"The cost is that you cannot precompute anything. Scoring 50 passages is 50 forward passes at query time, so a cross-encoder cannot search a corpus; it can only re-order a shortlist."}]},{"t":"p","runs":[{"text":"Hence the standard shape: "},{"text":"retrieve 50 cheaply, rerank them, keep 5.","b":true}]},{"t":"h2","id":"what-it-is-worth","runs":[{"text":"What it is worth"}]},{"t":"p","runs":[{"text":"Reranking is usually the highest-return single addition to a retrieval pipeline, ahead of a better embedding model and well ahead of prompt tuning. The reason is structural: your retriever’s job becomes recall at 50, which is a much easier target than precision at 5, and the reranker supplies the precision."}]},{"t":"p","runs":[{"text":"It also directly serves the budget. The k-curve lesson showed accuracy peaking and falling as passages are added. A reranker lets you sit at the good end of that curve: five passages that are the best five, rather than ten hedged ones."}]},{"t":"h2","id":"the-free-path-is-competitive-here","runs":[{"text":"The free path is competitive here"}],"role":"free"},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import CrossEncoder\nce = CrossEncoder(\"BAAI/bge-reranker-base\")          # ~280MB, CPU is fine\npairs = [(query, c.text) for c in candidates]\nfor c, s in zip(candidates, ce.predict(pairs)):\n    c.rerank_score = float(s)\ncandidates.sort(key=lambda c: c.rerank_score, reverse=True)"},{"t":"p","runs":[{"text":"On a laptop CPU, scoring 50 short passages with a small cross-encoder takes on the order of a few hundred milliseconds; on a modest GPU it is tens of milliseconds. "},{"text":"bge-reranker-v2-m3","code":true},{"text":" is the multilingual option and handles Indic and East Asian languages far better than English-only models: worth the extra size if your users do not all write in English."}]},{"t":"p","runs":[{"text":"The paid services (Cohere Rerank, Voyage, Jina) are strong, sometimes better, and priced per thousand documents scored. They are worth the money when latency at scale matters more than the bill. They are not required to learn or to ship this."}]},{"t":"h2","id":"the-truncation-trap","runs":[{"text":"The truncation trap"}]},{"t":"p","runs":[{"text":"The detail that catches almost everybody: "},{"text":"cross-encoders have a short maximum input","b":true},{"text":", often 512 tokens for query and passage combined. A 900-token chunk gets silently truncated, and the reranker scores only its first half."}]},{"t":"p","runs":[{"text":"So a passage whose relevant sentence is near the end scores as though that sentence does not exist, and it drops out of your top five. The symptom is a reranker that seems to make things worse on long chunks, which teams then blame on the reranker rather than on the truncation."}]},{"t":"p","runs":[{"text":"Three fixes. Keep chunks inside the reranker’s window at ingestion. Score the chunk in overlapping halves and take the maximum. Or use a long-context reranker and pay for it in latency. Check your model’s actual limit rather than assuming; it is one line in the model card and it changes the design."}]},{"t":"h2","id":"rerank-scores-are-worth-keeping","runs":[{"text":"Rerank scores are worth keeping"}]},{"t":"p","runs":[{"text":"Unlike a cosine similarity, a cross-encoder score is trained on relevance labels, so it means something closer to what you want. Two uses beyond ordering."}]},{"t":"p","runs":[{"text":"Abstention.","b":true},{"text":" Calibrate a floor on judged examples, and when the top reranked score falls below it, return “I do not have this” without a generation call. This is a better abstention signal than the embedding score gap from the earlier lesson, for the obvious reason."}]},{"t":"p","runs":[{"text":"Cutting adaptively.","b":true},{"text":" Instead of always taking five, take everything above the floor up to the token budget. Easy questions take two passages; hard ones take eight. Do not put the score in the context: the metadata lesson explains why."}]},{"t":"h2","id":"where-reranking-does-not-help","runs":[{"text":"Where reranking does not help"}]},{"t":"p","runs":[{"text":"It cannot retrieve what the first stage missed. If your gold passage is not in the 50, the reranker will never see it, and improving the reranker is wasted effort. Measure recall at 50 first: if it is 70%, your ceiling is 70% however good the reranker is, and the work belongs in the first stage."}]},{"t":"p","runs":[{"text":"It also adds a serial step to the request path. On a latency-critical surface (typeahead, an in-editor suggestion) a few hundred milliseconds may cost more than the quality is worth. Reranking is a quality-for-latency trade and it should be made deliberately, not because a tutorial included it."}]}],"check":{"question":[{"text":"A pipeline retrieves 50 candidates with recall@50 of 71%, adds a strong reranker, and end-to-end accuracy barely moves. What does this indicate?"}],"options":[{"text":[{"text":"The reranker’s scores are uncalibrated, so the cut-off is admitting weak passages."}],"correct":false,"tempting":[{"text":"Calibration affects where you cut, not whether the correct passage is available to be ranked at all."}]},{"text":[{"text":"The first stage misses the correct passage on 29% of queries, and reranking cannot recover what was never retrieved."}],"correct":true,"tempting":null},{"text":[{"text":"The reranker and the embedding model disagree, so their orderings cancel out."}],"correct":false,"tempting":[{"text":"Treats reranking as a vote when it wholly replaces the ordering of the shortlist it is given."}]},{"text":[{"text":"Reranking only helps when k is large, and five passages is too few for it to matter."}],"correct":false,"tempting":[{"text":"Inverts the relationship: reranking matters most precisely when few passages will be kept."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"keywords-and-vectors-together","title":"Keywords still win, so run both","minutes":9},"next":{"slug":"structured-sources-in-context","title":"Rows, records and schemas in the window","minutes":9}}}