Reranking, and how many chunks to actually send

Two kinds of encoder

Everything so far has used a bi-encoder: the query is embedded on its own, each passage was embedded on its own months ago, and they meet at a dot product. That separation is what makes search fast, because the passages are pre-computed. It is also why the ranking is approximate. The model that embedded the passage never saw the question. It encoded what the passage is about, and “about the same thing as the question” is not “answers the question”.

A cross-encoder takes the query and one passage together as a single input and runs attention across both. Every word of the question can attend to every word of the passage, so the model can judge whether this passage resolves that question rather than whether they share a topic. The price is that nothing can be pre-computed. Scoring is one forward pass per query-passage pair, at query time, and it is ten to a hundred times slower than a dot product.

So the standard pipeline is a funnel. Retrieve fifty candidates cheaply with the bi-encoder and BM25. Rerank those fifty with the cross-encoder. Keep five.

The free path, and the cost

Open cross-encoders: bge-reranker-v2-m3 handles many languages; the ms-marco-MiniLM rerankers are small and fast. On a CPU a small reranker scores a pair in roughly 20 to 50 milliseconds, so fifty pairs is one to two seconds: too slow for interactive search on a CPU, fine on a modest GPU where the batch scores in under 100 milliseconds. Hosted: Cohere and Voyage sell reranking at around a dollar or two per thousand queries.

A language model can also rerank by being shown the candidates and asked to order them. It is the most accurate option and the most expensive, and it is what you reach for on hard, low-volume queries.

What reranking buys

In benchmark comparisons a cross-encoder over a decent first stage adds roughly ten to twenty points of ranking quality, and the gain is largest exactly when the first stage is weakest: a small embedding model, a noisy corpus, long passages. It is also the stage that lets you set an honest cutoff, because cross-encoder scores separate “relevant” from “not” far more cleanly than cosine similarity does. Calibrate the cutoff on your labelled queries; below it, the answer is “nothing in the corpus addresses this”, and the model should be told so rather than handed five weak passages to improvise from.

How many chunks to send

The instinct is to send as many as fit, on the theory that more context cannot hurt. It can, for three separate reasons.

Every passage costs tokens on every query. Five chunks of 400 tokens is 2,000 tokens of input per request; at ten thousand queries a day that is twenty million tokens a day, before the conversation itself.

Irrelevant passages are distractors. A model given three relevant and seven irrelevant chunks answers worse than one given the three alone, because attention is spread across text that looks pertinent and is not.

Position matters. Models attend more reliably to the start and end of a long context than to its middle, so the eighth of twelve chunks is the least likely to be used, whatever its relevance.

Measure it on your evaluation set: answer quality at three, five, eight and twelve chunks. For factual question-answering the curve typically flattens somewhere around five to eight. For summarising across sources it keeps rising longer. Send the number where your curve flattens, and put the highest-ranked chunk first and the second-highest last.

Diversity

The top five by relevance are often five near-copies: a paragraph that appears in the FAQ, the manual, a release note and two support articles. A budget of five spent on one fact is a budget of one. Maximal marginal relevance penalises a candidate for its similarity to chunks already chosen, and a single pass of it over the reranked list (pick the top, then repeatedly pick the candidate with the best relevance minus overlap) turns five copies into five facts. Deduplicate at ingestion too, but expect duplicates to survive.

The latency budget

Write it down before you build:

• Embed the query: 5 to 100 milliseconds, local or hosted.
• Vector and keyword search: 5 to 50 milliseconds indexed.
• Rerank fifty candidates: 100 milliseconds on a GPU, a second or more on a CPU.
• Generate the answer: one to three seconds.

Reranking is the stage most likely to blow the budget, and the honest fix is not to remove it but to rerank twenty candidates instead of fifty, or to run the small reranker on a GPU. A retrieval stage that returns in 30 milliseconds and hands the model the wrong passages has saved nothing.