Hybrid search: why keyword search is still half the answer

The queries an embedding fails

Open the search logs of any real product and a large share of the queries are not questions. They are error E-4471, INV-2024-00931, Priya Raghunathan, a SKU, a function name, a postcode. The last lesson on embeddings explained why these blur: the model breaks a rare string into subword pieces that carry almost no consistent meaning, so E-4471 and E-4417 land in nearly the same place.

Keyword search has the opposite blind spot. It cannot see that “cancel my plan” and “stop the subscription” mean the same thing. It has been the backbone of search for thirty years precisely because on exact terms it is nearly perfect.

A production system needs both, and the standard name for both is hybrid search.

BM25 in one paragraph

BM25 scores a document for a query by adding up, for each query term, how often the term appears in the document, scaled down by how common the term is across the whole corpus, and normalised for document length so that long documents do not win by volume. Rare terms count for more, which is exactly why it finds an invoice number: that string occurs in one document and nowhere else. Two parameters, k1 and b, control the saturation of term frequency and the strength of length normalisation, and the defaults are fine.

The free path is everywhere. Postgres has full-text search built in through tsvector and ts_rank. SQLite has FTS5. Elasticsearch and OpenSearch are free to run. For a corpus that fits in memory, the Python package rank_bm25 is a few lines:

from rank_bm25 import BM25Okapi
bm25 = BM25Okapi([doc.split() for doc in docs])
scores = bm25.get_scores(query.split())

Fusing two ranked lists

You now have two lists for one query, one from the vector index and one from BM25, and the wrong way to combine them is to add the scores. A BM25 score is unbounded and depends on the corpus; a cosine score sits between zero and one. Adding them lets whichever has the larger range decide everything.

Reciprocal rank fusion ignores the scores and uses only the positions. A document’s fused score is the sum, over every list it appears in, of one divided by a constant plus its rank in that list. The constant is conventionally 60, and it exists so that the difference between first and second place is not overwhelming.

def rrf(*ranked_lists, k=60):
    fused = {}
    for lst in ranked_lists:
        for rank, doc_id in enumerate(lst, start=1):
            fused[doc_id] = fused.get(doc_id, 0) + 1 / (k + rank)
    return sorted(fused, key=fused.get, reverse=True)

A document that is fifth in both lists outranks one that is first in one list and absent from the other, which is usually what you want: agreement between two different signals is strong evidence.

The gain you should expect

In published comparisons across mixed corpora, and in most teams’ own tests, hybrid retrieval beats either method alone by somewhere between five and fifteen points of recall at ten. The gain is small on corpora of plain-language questions and answers, where the embedding was already doing well, and large on anything full of identifiers: code, logs, catalogues, legal references, medical codes. If your queries include even one identifier in ten, hybrid pays for itself on that tenth alone.

Tokenisation is where hybrid breaks

BM25 matches tokens, and a token is whatever your analyser decided it is. Three failures to check.

An English stemmer applied to Hindi does nothing useful, and the default english configuration in Postgres will also drop words it considers stop words. For Indian languages use the simple configuration or a language-specific analyser, and test with real queries.

Code needs splitting on camelCase and snake_case; otherwise getUserById is one token that matches only itself.

Identifiers with punctuation get split unpredictably: INV-2024-00931 may become three tokens, of which 2024 matches ten thousand documents. Index a normalised copy of identifier-like strings with the punctuation removed, so the whole string is one rare token.

Weighting one side

When a query looks like an identifier (a regular expression for digits, uppercase codes and hyphens does the job) the keyword list deserves more weight. RRF accepts a weight per list; giving BM25 double weight on such queries, and the vector list double weight on long natural-language questions, is a cheap improvement that a small classifier can later refine.

The honest limits

Hybrid search does not fix a corpus that lacks the answer, and it does not fix chunks that were cut mid-sentence. It also doubles the index maintenance: two indexes that must be updated together, and a delete that reaches one but not the other leaves a ghost document that keeps appearing in half the results. Put both updates in one transaction where the storage allows it, and where it does not, run a nightly reconciliation that compares the two document sets.