The second pass that fixes the first

Two architectures, and why both exist

A bi-encoder (everything so far) encodes the query and the document separately and compares vectors. The document’s vector is computed once, in advance, so searching a million documents is a million dot products. Fast, and the comparison is crude, because the model never saw the query and the document together.

A cross-encoder takes the query and the document as one input and outputs a single relevance score:

from sentence_transformers import CrossEncoder

ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
print(ce.predict([("where is my parcel", "Your order shipped on Tuesday."),
                  ("where is my parcel", "How to change your password.")]))
# [ 8.21 -6.94 ]

Every query token can attend to every document token, so the model can tell that a document answers a question rather than merely being about the same topic. That is exactly the weakness the first lesson named.

The cost: nothing can be precomputed. Scoring one query against a million documents means a million forward passes. It is completely impractical as a search method.

The pattern: retrieve then rerank

candidates = search(query, k=50)                      # bi-encoder, milliseconds
pairs = [(query, c["text"]) for c in candidates]
scores = ce.predict(pairs)                            # 50 forward passes
best = [candidates[i] for i in scores.argsort()[::-1][:5]]

Fifty forward passes on a small cross-encoder is roughly 50 to 200 ms on a CPU and a few milliseconds on a GPU. You get most of the cross-encoder’s accuracy at a tiny fraction of its cost, because the expensive model only ever sees fifty documents.

The gains are large and well replicated: on standard retrieval benchmarks, reranking the top 50 commonly improves precision at 5 by ten to twenty points over the bi-encoder alone. Very few single changes in this area do that.

Choosing how many to retrieve

The reranker can only reorder what it is given. Retrieve 10 and the right answer at rank 40 is lost forever; retrieve 200 and latency grows linearly.

The number to measure is your bi-encoder’s recall@k: how often the correct document appears in the top k at all. Retrieve at the k where recall plateaus, typically 30 to 100. If recall@50 is 0.92 and recall@200 is 0.93, retrieving 200 buys you almost nothing and quadruples the rerank cost.

This is why the thirty-question set exists. Without it, both numbers are guesses.

Which reranker

• cross-encoder/ms-marco-MiniLM-L-6-v2: 90 MB, English, fast, the sensible default.
• BAAI/bge-reranker-base and -large: stronger, multilingual coverage including Chinese and several Indic languages.
• jina-reranker family: long inputs.

All run on CPU. A reranker is usually smaller than the generative model it feeds, and it is the cheapest quality improvement in a retrieval pipeline.

The language-model reranker, and its trade

You can also ask a general instruction model to rate relevance from 0 to 10. It works, needs no extra model if you already have one, and is slower, more expensive and less consistent than a trained cross-encoder: the same prompt on the same pair can score differently across runs unless you fix the sampling. Use it when adding a dependency is impossible; prefer the trained model otherwise.

What reranking cannot fix

• A document that was never retrieved. If chunking lost the fact, or the embedding model does not handle your language, no reranker helps. Fix retrieval first.
• A document that does not exist. The corpus is the ceiling.
• Diversity. A reranker will happily put five near-identical chunks at the top. Apply maximal marginal relevance after reranking, not before.

Latency

A full pipeline (embed query, search, rerank 50) is roughly 5 ms to embed, 10 ms to search, 100 ms to rerank on CPU. Around 120 ms, before any generation. That is acceptable behind a web request and noticeable in a tight loop, and the rerank step dominates, so it is where you tune.

Do this now

Run your thirty questions with and without reranking and compare recall@1. This is normally the largest single improvement available in a retrieval system, and it takes twenty lines.