Keywords still win, so run both

What each method is good at

Dense vector search finds text that means something similar. Sparse keyword search finds text that contains the same words. They fail in complementary ways, which is why hybrid retrieval is not a compromise but the default.

Keyword search (BM25 in practice) wins decisively on:

• Identifiers. ORD-88213, ECONNREFUSED, Section 80C, a part number, a person’s surname. These are rare tokens; rarity is exactly what BM25 rewards and exactly what a dense embedding smooths away.
• Exact phrases the user is quoting from something they are looking at.
• Domain jargon the embedding model never saw. Product names, internal acronyms, terms coined after the embedding model was trained.
• Negation and precise wording, indirectly: the words are matched literally, so “not” is a token like any other rather than a nuance the embedding blurs.

Dense search wins on:

• Vocabulary mismatch. “My money hasn’t come back” against “reimbursement processing timelines”.
• Conceptual questions with no distinctive shared terms.
• Cross-lingual retrieval, with a multilingual embedding model: a Hindi question can retrieve an English document, which BM25 cannot do at all.

A system with only one of them has a predictable hole, and users find it in the first week.

BM25 in one paragraph

BM25 scores a document by, for each query term, how often it appears in the document, damped so the tenth occurrence adds little, multiplied by how rare the term is across the corpus, and divided down if the document is long relative to average. That is the entire idea. Its strength is the rarity weighting: a match on ECONNREFUSED counts for enormously more than a match on error, without anybody configuring that.

Free implementations everywhere. rank_bm25 in Python for small corpora, SQLite’s FTS5 for anything up to a few million documents on a single machine, PostgreSQL’s tsvector if your data already lives there, OpenSearch or Elasticsearch at scale. None of this needs a GPU or a hosted service.

Fusing the two lists

You have two ranked lists with incomparable scores: a cosine of 0.71 and a BM25 of 14.3. Do not try to add them. Score normalisation across methods is fragile: it depends on the query, the corpus and the day.

Reciprocal rank fusion ignores the scores and uses only the positions:

def rrf(lists, k=60):
    scores = defaultdict(float)
    for lst in lists:
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

The constant k=60 is the value from the original paper and it works well enough that almost nobody changes it. Its job is to flatten the difference between the top few positions so that one list cannot dominate on rank alone; larger values flatten more.

RRF is robust, needs no tuning, and composes with any number of lists: vector, keyword, one per rewritten query, one per sub-question. Its limitation is the flip side of its strength: it throws away magnitude, so a query where one method found an obviously perfect match gets no extra credit for that. Where you have a reranker downstream this does not matter much, because the reranker restores a meaningful score over the fused candidates.

Getting the tokenisation right for your languages

BM25 needs words, and word boundaries are a language-specific problem that default configurations quietly get wrong.

English stemming is standard and helps. Hindi, Bengali and Tamil are heavily inflected and need their own analysers or the same word in two forms will not match. Chinese, Japanese and Thai have no spaces at all and require a segmenter: without one, a whitespace tokeniser produces one enormous token per sentence and keyword search silently returns nothing.

Check this explicitly if you serve those languages. The failure is silent: no error, no warning, just a keyword index that never contributes to the fused results, and a hybrid system that is a dense-only system with extra latency.

Filters apply to both

Everything the earlier lesson said about pre-filtering applies to each retriever separately. A permission filter applied to only one arm of a hybrid search leaks through the other, and it is an easy mistake because the two arms are usually different pieces of infrastructure with different filter syntax.

Write one function that builds the filter and have both retrievers consume it, so the two cannot drift apart.

What to expect from adding it

Adding BM25 to a dense-only system typically produces a solid recall improvement, concentrated entirely in a specific slice of queries: the ones containing identifiers, names and rare terms. On conceptual questions it changes almost nothing.

Which means the headline average understates it. Report it split by query type, because “3 points of recall” hides the fact that a category of queries went from failing outright to working, and those are usually the queries where a user was staring at an error code and knew the answer existed.