Training a reranker

Why a cross-encoder is better

An embedding model encodes the query and the passage separately, into two vectors that are compared afterwards. Whatever the passage says about the query has to survive being compressed into a fixed vector before the query is ever seen.

A cross-encoder puts both into the model at once ([query] [SEP] [passage]) and produces one relevance score. Every token of the query can attend to every token of the passage. That is why rerankers beat bi-encoders consistently, often by a wide margin on the harder queries.

The price is that the score cannot be precomputed. A bi-encoder embeds your corpus once, and a query is one vector comparison against an index. A cross-encoder must run the model once per candidate, at query time.

The two-stage architecture

So you use both:

1. Retrieve 50 to 100 candidates with the bi-encoder plus BM25. Fast, over the whole corpus.
2. Rerank those candidates with the cross-encoder. Slow per item, but only 50 items.
3. Pass the top 5 to the generator.

The arithmetic that makes this work: a 22M-parameter cross-encoder scores 50 query-passage pairs in roughly 15 to 40 milliseconds on a GPU, or a few hundred milliseconds on a CPU. Doing the same over a 200,000-document corpus would take minutes. The two-stage design buys cross-encoder quality at bi-encoder latency.

Training it

from sentence_transformers import CrossEncoder
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", num_labels=1)

The data is triples of query, passage and label. With binary cross-entropy the label is 1 for relevant and 0 for not, and the ratio matters. Around four negatives per positive is a common starting point; far more negatives and the model learns to say “no” to everything, which scores well on accuracy and ranks nothing.

The negatives should be mined from your retriever’s actual output, as with embedding training. A reranker’s entire job is to sort the candidates your first stage returns, so training it on random negatives trains it for a task it will never face.

The listwise alternative

Binary labels treat each pair independently, but ranking is about order. Losses that compare a positive against several negatives for the same query within one forward group (the family that recent reranker work uses) optimise the ordering directly and generally produce better rankings from the same data. CachedMultipleNegativesRankingLoss has a cross-encoder counterpart in recent sentence-transformers versions, and it is worth preferring where available.

Evaluating a reranker properly

The metric is the ranking quality of the reranked list, not classification accuracy on pairs. NDCG at 10 and MRR at 10 are the standard measures.

The honest comparison is against your first stage, on the same candidates:

stage 1 (bi-encoder + BM25), NDCG@10 = 0.61
+ off-the-shelf reranker,    NDCG@10 = 0.74
+ fine-tuned reranker,       NDCG@10 = 0.79

Note what that shows. The off-the-shelf reranker delivered most of the gain, and fine-tuning added a smaller increment. Try the pretrained reranker before training one. The public cross-encoders are trained on very large relevance datasets and are strong out of the box; on many corpora they are good enough, and you have spent nothing.

The latency budget

The decision that constrains everything is how many candidates you can afford to rerank.

• 22M parameters, 50 candidates: tens of milliseconds on a GPU. Comfortable.
• 278M parameters, 50 candidates: low hundreds of milliseconds. Noticeable in an interactive product.
• A large LLM-based reranker: hundreds of milliseconds to seconds. Only for offline or high-value queries.

Halving the candidate count halves the cost, and the recall you lose by reranking 25 instead of 50 is usually small: measure it, because it is the cheapest latency win in the pipeline.

Where it belongs in the course

Back in module one the argument was that retrieval should be fixed before the generator. This lesson and the previous one are what that means concretely. A fine-tuned embedding model raises what can be found; a reranker raises what reaches the top. Together they typically cost a day and a few hundred megabytes, and they move end-to-end quality further than a generator fine-tune costing ten times as much.