{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"reranking-and-how-many-chunks","title":"Reranking, and how many chunks to actually send","url":"https://addaly.com/learn/building-with-ai/reranking-and-how-many-chunks","position":38,"of":80,"minutes":9,"answer":"A cross-encoder reads the query and passage together and so can judge whether the passage answers the question, which a bi-encoder's separate vectors never can.","keyIdea":"A cross-encoder reads the query and passage together and so can judge whether the passage answers the question, which a bi-encoder's separate vectors never can, and the number of chunks you then send is a measured trade between recall, distraction and tokens per query rather than \"as many as fit\"."},"body":{"format":"markdown","text":"## Two kinds of encoder\n\nEverything so far has used a bi-encoder: the query is embedded on its own, each passage was embedded on its own months ago, and they meet at a dot product. That separation is what makes search fast, because the passages are pre-computed. It is also why the ranking is approximate. The model that embedded the passage never saw the question. It encoded what the passage is about, and \"about the same thing as the question\" is not \"answers the question\".\n\nA cross-encoder takes the query and one passage together as a single input and runs attention across both. Every word of the question can attend to every word of the passage, so the model can judge whether this passage resolves that question rather than whether they share a topic. The price is that nothing can be pre-computed. Scoring is one forward pass per query-passage pair, at query time, and it is ten to a hundred times slower than a dot product.\n\nSo the standard pipeline is a funnel. Retrieve fifty candidates cheaply with the bi-encoder and BM25. Rerank those fifty with the cross-encoder. Keep five.\n\n## The free path, and the cost\n\nOpen cross-encoders: `bge-reranker-v2-m3` handles many languages; the `ms-marco-MiniLM` rerankers are small and fast. On a CPU a small reranker scores a pair in roughly 20 to 50 milliseconds, so fifty pairs is one to two seconds: too slow for interactive search on a CPU, fine on a modest GPU where the batch scores in under 100 milliseconds. Hosted: Cohere and Voyage sell reranking at around a dollar or two per thousand queries.\n\nA language model can also rerank by being shown the candidates and asked to order them. It is the most accurate option and the most expensive, and it is what you reach for on hard, low-volume queries.\n\n## What reranking buys\n\nIn benchmark comparisons a cross-encoder over a decent first stage adds roughly ten to twenty points of ranking quality, and the gain is largest exactly when the first stage is weakest: a small embedding model, a noisy corpus, long passages. It is also the stage that lets you set an honest cutoff, because cross-encoder scores separate \"relevant\" from \"not\" far more cleanly than cosine similarity does. Calibrate the cutoff on your labelled queries; below it, the answer is \"nothing in the corpus addresses this\", and the model should be told so rather than handed five weak passages to improvise from.\n\n## How many chunks to send\n\nThe instinct is to send as many as fit, on the theory that more context cannot hurt. It can, for three separate reasons.\n\nEvery passage costs tokens on every query. Five chunks of 400 tokens is 2,000 tokens of input per request; at ten thousand queries a day that is twenty million tokens a day, before the conversation itself.\n\nIrrelevant passages are distractors. A model given three relevant and seven irrelevant chunks answers worse than one given the three alone, because attention is spread across text that looks pertinent and is not.\n\nPosition matters. Models attend more reliably to the start and end of a long context than to its middle, so the eighth of twelve chunks is the least likely to be used, whatever its relevance.\n\nMeasure it on your evaluation set: answer quality at three, five, eight and twelve chunks. For factual question-answering the curve typically flattens somewhere around five to eight. For summarising across sources it keeps rising longer. Send the number where your curve flattens, and put the highest-ranked chunk first and the second-highest last.\n\n## Diversity\n\nThe top five by relevance are often five near-copies: a paragraph that appears in the FAQ, the manual, a release note and two support articles. A budget of five spent on one fact is a budget of one. Maximal marginal relevance penalises a candidate for its similarity to chunks already chosen, and a single pass of it over the reranked list (pick the top, then repeatedly pick the candidate with the best relevance minus overlap) turns five copies into five facts. Deduplicate at ingestion too, but expect duplicates to survive.\n\n## The latency budget\n\nWrite it down before you build:\n\n- Embed the query: 5 to 100 milliseconds, local or hosted.\n- Vector and keyword search: 5 to 50 milliseconds indexed.\n- Rerank fifty candidates: 100 milliseconds on a GPU, a second or more on a CPU.\n- Generate the answer: one to three seconds.\n\nReranking is the stage most likely to blow the budget, and the honest fix is not to remove it but to rerank twenty candidates instead of fifty, or to run the small reranker on a GPU. A retrieval stage that returns in 30 milliseconds and hands the model the wrong passages has saved nothing.\n\n> Figure: More context is not more accuracy. Line chart of Questions answered correctly, per cent against Chunks sent to the model: Factual question answering (1, 58) (3, 71) (5, 79) (8, 80) (12, 76) (20, 69). Measured on 120 gold questions after reranking. The curve flattens near five to eight and falls at twelve, because irrelevant passages are distractors and the middle of a long context is attended to least. Each extra chunk is another 400 tokens on every query, forever."},"check":{"question":"Why cannot a cross-encoder's scores be pre-computed and stored the way passage embeddings are?","options":[{"text":"Cross-encoders produce a single score rather than a vector, and a plain score cannot be placed in an index and searched against later.","correct":false,"tempting":"Scores can be stored as easily as vectors; the obstacle is what the score depends on, not its type."},{"text":"A cross-encoder's score depends on the query and the passage together, so it can only be computed once the query is known.","correct":true},{"text":"Cross-encoders are too large to run at ingestion time.","correct":false,"tempting":"Size affects cost, not possibility; a large model can be run offline if its output were reusable."},{"text":"Cross-encoder outputs change between runs, so a stored score would be stale.","correct":false,"tempting":"The model is deterministic; the score is stable for a given pair, it is simply undefined without the query."}]},"prev":{"slug":"hybrid-search-keywords-still-win","title":"Hybrid search: why keyword search is still half the answer","url":"https://addaly.com/learn/building-with-ai/hybrid-search-keywords-still-win"},"next":{"slug":"rewriting-the-question","title":"Rewriting the question before you search it","url":"https://addaly.com/learn/building-with-ai/rewriting-the-question"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reranking, and how many chunks to actually send', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/reranking-and-how-many-chunks (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reranking, and how many chunks to actually send","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"A cross-encoder reads the query and passage together and so can judge whether the passage answers the question, which a bi-encoder’s separate vectors never can, and the number of chunks you then send is a measured trade between recall, distraction and tokens per query rather than “as many as fit”.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"More context is not more accuracy","series":[{"label":"Factual question answering","points":[[1,58],[3,71],[5,79],[8,80],[12,76],[20,69]]}],"xLabel":"Chunks sent to the model","yLabel":"Questions answered correctly, per cent","caption":"Measured on 120 gold questions after reranking. The curve flattens near five to eight and falls at twelve, because irrelevant passages are distractors and the middle of a long context is attended to least. Each extra chunk is another 400 tokens on every query, forever."},"alt":"Figure: More context is not more accuracy. Line chart of Questions answered correctly, per cent against Chunks sent to the model: Factual question answering (1, 58) (3, 71) (5, 79) (8, 80) (12, 76) (20, 69). Measured on 120 gold questions after reranking. The curve flattens near five to eight and falls at twelve, because irrelevant passages are distractors and the middle of a long context is attended to least. Each extra chunk is another 400 tokens on every query, forever."}},"blocks":[{"t":"h2","id":"two-kinds-of-encoder","runs":[{"text":"Two kinds of encoder"}]},{"t":"p","runs":[{"text":"Everything so far has used a bi-encoder: the query is embedded on its own, each passage was embedded on its own months ago, and they meet at a dot product. That separation is what makes search fast, because the passages are pre-computed. It is also why the ranking is approximate. The model that embedded the passage never saw the question. It encoded what the passage is about, and “about the same thing as the question” is not “answers the question”."}]},{"t":"p","runs":[{"text":"A cross-encoder takes the query and one passage together as a single input and runs attention across both. Every word of the question can attend to every word of the passage, so the model can judge whether this passage resolves that question rather than whether they share a topic. The price is that nothing can be pre-computed. Scoring is one forward pass per query-passage pair, at query time, and it is ten to a hundred times slower than a dot product."}]},{"t":"p","runs":[{"text":"So the standard pipeline is a funnel. Retrieve fifty candidates cheaply with the bi-encoder and BM25. Rerank those fifty with the cross-encoder. Keep five."}]},{"t":"h2","id":"the-free-path-and-the-cost","runs":[{"text":"The free path, and the cost"}],"role":"free"},{"t":"p","runs":[{"text":"Open cross-encoders: "},{"text":"bge-reranker-v2-m3","code":true},{"text":" handles many languages; the "},{"text":"ms-marco-MiniLM","code":true},{"text":" rerankers are small and fast. On a CPU a small reranker scores a pair in roughly 20 to 50 milliseconds, so fifty pairs is one to two seconds: too slow for interactive search on a CPU, fine on a modest GPU where the batch scores in under 100 milliseconds. Hosted: Cohere and Voyage sell reranking at around a dollar or two per thousand queries."}]},{"t":"p","runs":[{"text":"A language model can also rerank by being shown the candidates and asked to order them. It is the most accurate option and the most expensive, and it is what you reach for on hard, low-volume queries."}]},{"t":"h2","id":"what-reranking-buys","runs":[{"text":"What reranking buys"}]},{"t":"p","runs":[{"text":"In benchmark comparisons a cross-encoder over a decent first stage adds roughly ten to twenty points of ranking quality, and the gain is largest exactly when the first stage is weakest: a small embedding model, a noisy corpus, long passages. It is also the stage that lets you set an honest cutoff, because cross-encoder scores separate “relevant” from “not” far more cleanly than cosine similarity does. Calibrate the cutoff on your labelled queries; below it, the answer is “nothing in the corpus addresses this”, and the model should be told so rather than handed five weak passages to improvise from."}]},{"t":"h2","id":"how-many-chunks-to-send","runs":[{"text":"How many chunks to send"}]},{"t":"p","runs":[{"text":"The instinct is to send as many as fit, on the theory that more context cannot hurt. It can, for three separate reasons."}]},{"t":"p","runs":[{"text":"Every passage costs tokens on every query. Five chunks of 400 tokens is 2,000 tokens of input per request; at ten thousand queries a day that is twenty million tokens a day, before the conversation itself."}]},{"t":"p","runs":[{"text":"Irrelevant passages are distractors. A model given three relevant and seven irrelevant chunks answers worse than one given the three alone, because attention is spread across text that looks pertinent and is not."}]},{"t":"p","runs":[{"text":"Position matters. Models attend more reliably to the start and end of a long context than to its middle, so the eighth of twelve chunks is the least likely to be used, whatever its relevance."}]},{"t":"p","runs":[{"text":"Measure it on your evaluation set: answer quality at three, five, eight and twelve chunks. For factual question-answering the curve typically flattens somewhere around five to eight. For summarising across sources it keeps rising longer. Send the number where your curve flattens, and put the highest-ranked chunk first and the second-highest last."}]},{"t":"h2","id":"diversity","runs":[{"text":"Diversity"}]},{"t":"p","runs":[{"text":"The top five by relevance are often five near-copies: a paragraph that appears in the FAQ, the manual, a release note and two support articles. A budget of five spent on one fact is a budget of one. Maximal marginal relevance penalises a candidate for its similarity to chunks already chosen, and a single pass of it over the reranked list (pick the top, then repeatedly pick the candidate with the best relevance minus overlap) turns five copies into five facts. Deduplicate at ingestion too, but expect duplicates to survive."}]},{"t":"h2","id":"the-latency-budget","runs":[{"text":"The latency budget"}]},{"t":"p","runs":[{"text":"Write it down before you build:"}]},{"t":"ul","items":[[{"text":"Embed the query: 5 to 100 milliseconds, local or hosted."}],[{"text":"Vector and keyword search: 5 to 50 milliseconds indexed."}],[{"text":"Rerank fifty candidates: 100 milliseconds on a GPU, a second or more on a CPU."}],[{"text":"Generate the answer: one to three seconds."}]]},{"t":"p","runs":[{"text":"Reranking is the stage most likely to blow the budget, and the honest fix is not to remove it but to rerank twenty candidates instead of fifty, or to run the small reranker on a GPU. A retrieval stage that returns in 30 milliseconds and hands the model the wrong passages has saved nothing."}]}],"check":{"question":[{"text":"Why cannot a cross-encoder’s scores be pre-computed and stored the way passage embeddings are?"}],"options":[{"text":[{"text":"Cross-encoders produce a single score rather than a vector, and a plain score cannot be placed in an index and searched against later."}],"correct":false,"tempting":[{"text":"Scores can be stored as easily as vectors; the obstacle is what the score depends on, not its type."}]},{"text":[{"text":"A cross-encoder’s score depends on the query and the passage together, so it can only be computed once the query is known."}],"correct":true,"tempting":null},{"text":[{"text":"Cross-encoders are too large to run at ingestion time."}],"correct":false,"tempting":[{"text":"Size affects cost, not possibility; a large model can be run offline if its output were reusable."}]},{"text":[{"text":"Cross-encoder outputs change between runs, so a stored score would be stale."}],"correct":false,"tempting":[{"text":"The model is deterministic; the score is stable for a given pair, it is simply undefined without the query."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"hybrid-search-keywords-still-win","title":"Hybrid search: why keyword search is still half the answer","minutes":9},"next":{"slug":"rewriting-the-question","title":"Rewriting the question before you search it","minutes":9}}}