Reranking, and why it reads better than the retriever

Two ways to compare a query with a passage

A bi-encoder (an ordinary embedding model) encodes the query and the passage separately, into two vectors, and compares them. Separately is the key word: the passage’s vector was computed months ago with no knowledge of any query. That is what makes it fast enough to search millions of documents, and it is also what limits it. All the interaction between the question and the text has to be squeezed through a single dot product between two summaries.

A cross-encoder puts the query and the passage into the model together and outputs one relevance score. Every token of the question can attend to every token of the passage. It can notice that the passage answers a different question about the same subject, that it contains a negation, that the number it gives is for a different plan.

The cost is that you cannot precompute anything. Scoring 50 passages is 50 forward passes at query time, so a cross-encoder cannot search a corpus; it can only re-order a shortlist.

Hence the standard shape: retrieve 50 cheaply, rerank them, keep 5.

What it is worth

Reranking is usually the highest-return single addition to a retrieval pipeline, ahead of a better embedding model and well ahead of prompt tuning. The reason is structural: your retriever’s job becomes recall at 50, which is a much easier target than precision at 5, and the reranker supplies the precision.

It also directly serves the budget. The k-curve lesson showed accuracy peaking and falling as passages are added. A reranker lets you sit at the good end of that curve: five passages that are the best five, rather than ten hedged ones.

The free path is competitive here

from sentence_transformers import CrossEncoder
ce = CrossEncoder("BAAI/bge-reranker-base")          # ~280MB, CPU is fine
pairs = [(query, c.text) for c in candidates]
for c, s in zip(candidates, ce.predict(pairs)):
    c.rerank_score = float(s)
candidates.sort(key=lambda c: c.rerank_score, reverse=True)

On a laptop CPU, scoring 50 short passages with a small cross-encoder takes on the order of a few hundred milliseconds; on a modest GPU it is tens of milliseconds. bge-reranker-v2-m3 is the multilingual option and handles Indic and East Asian languages far better than English-only models: worth the extra size if your users do not all write in English.

The paid services (Cohere Rerank, Voyage, Jina) are strong, sometimes better, and priced per thousand documents scored. They are worth the money when latency at scale matters more than the bill. They are not required to learn or to ship this.

The truncation trap

The detail that catches almost everybody: cross-encoders have a short maximum input, often 512 tokens for query and passage combined. A 900-token chunk gets silently truncated, and the reranker scores only its first half.

So a passage whose relevant sentence is near the end scores as though that sentence does not exist, and it drops out of your top five. The symptom is a reranker that seems to make things worse on long chunks, which teams then blame on the reranker rather than on the truncation.

Three fixes. Keep chunks inside the reranker’s window at ingestion. Score the chunk in overlapping halves and take the maximum. Or use a long-context reranker and pay for it in latency. Check your model’s actual limit rather than assuming; it is one line in the model card and it changes the design.

Rerank scores are worth keeping

Unlike a cosine similarity, a cross-encoder score is trained on relevance labels, so it means something closer to what you want. Two uses beyond ordering.

Abstention. Calibrate a floor on judged examples, and when the top reranked score falls below it, return “I do not have this” without a generation call. This is a better abstention signal than the embedding score gap from the earlier lesson, for the obvious reason.

Cutting adaptively. Instead of always taking five, take everything above the floor up to the token budget. Easy questions take two passages; hard ones take eight. Do not put the score in the context: the metadata lesson explains why.

Where reranking does not help

It cannot retrieve what the first stage missed. If your gold passage is not in the 50, the reranker will never see it, and improving the reranker is wasted effort. Measure recall at 50 first: if it is 70%, your ceiling is 70% however good the reranker is, and the work belongs in the first stage.

It also adds a serial step to the request path. On a latency-critical surface (typeahead, an in-editor suggestion) a few hundred milliseconds may cost more than the quality is worth. Reranking is a quality-for-latency trade and it should be made deliberately, not because a tutorial included it.