Rerankers, and why retrieval is two stages

The limitation that creates the second stage

An embedding model has to compress a document into a fixed vector before it knows what anybody will ask. That is what makes search fast: you embed a million documents once, and each query is a nearest-neighbour lookup. It is also what limits quality, because the compression happens without the query.

A reranker removes that limitation by paying for it. It takes the query and one document together, runs them through a transformer, and outputs a single relevance score. Every token of the query can attend to every token of the document. It is far more accurate and it is far too slow to run over a million documents.

So the standard shape is two stages:

1. Retrieve the top 50 to 100 candidates with the embedding model, in a few milliseconds.
2. Rerank those candidates with the cross-encoder, in tens to hundreds of milliseconds, and keep the top 5 to 10.

The vocabulary: the embedding model is a bi-encoder because it encodes each side separately; the reranker is a cross-encoder because it encodes them jointly.

What it is worth

On typical document collections, adding a reranker moves the quality of the top 5 results more than upgrading from a small embedding model to a large one, and it costs less to change. In a retrieval-augmented system, the top 5 is what the language model sees, so this is usually the highest-return single change available.

The mechanism explains where it helps most: cases where two candidates are topically identical but only one answers the question. The pair the payment succeeded and the payment failed from the previous lesson is exactly the case a bi-encoder cannot separate and a cross-encoder can, because the cross-encoder sees the word failed in the presence of the word the user actually asked about.

The open rerankers

• bge-reranker-v2-m3. Multilingual, permissive, the reliable default.
• Qwen3-Reranker. Several sizes, recent, strong.
• mxbai-rerank and jina-reranker are alternatives worth measuring against.
• ColBERT and its successors take a middle path called late interaction: they keep one vector per token rather than one per document, and compare token to token. More accurate than a bi-encoder, faster than a cross-encoder, and much larger indexes. Worth knowing exists; worth using when the index size is affordable.

from sentence_transformers import CrossEncoder
ce = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
scores = ce.predict([(query, doc) for doc in candidates])
top = [d for _, d in sorted(zip(scores, candidates), reverse=True)][:5]

A base-sized reranker scores roughly 50 pairs in a few hundred milliseconds on a CPU and far faster on a GPU. Budget for it: reranking 100 candidates costs more than retrieving them by a factor of ten or more.

The other half of the first stage

Dense retrieval has a specific blind spot: exact strings. Product codes, error numbers, surnames, an unusual acronym. The embedding compresses these into a general neighbourhood, and the exact match is not recoverable.

BM25 (classic keyword scoring, available in every search engine and in small libraries) has the opposite profile. It finds exact tokens and misses paraphrases.

Running both and combining them is called hybrid retrieval, and the usual combination method is reciprocal rank fusion, which merges two ranked lists by rank rather than by score, so you do not have to make two incomparable scores comparable:

score(d) = sum over lists of  1 / (60 + rank_of_d_in_list)

Hybrid plus a reranker is the configuration most production systems converge on, and it is not fashion; it is two different failure modes being covered by two different mechanisms.

The limit worth stating

A reranker cannot find what the first stage missed. It only reorders what it is given. If the correct document is not in the top 100 retrieved, no reranker will produce it, and every symptom will look like a reranker problem.

So measure the two stages separately. Recall at 100 for the retriever tells you whether the answer is even in the candidate set; that is the number to fix first. Then measure precision in the top 5 after reranking. Teams that report a single end-to-end number spend weeks tuning the wrong stage.