{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"hybrid-search-keywords-still-win","title":"Hybrid search: why keyword search is still half the answer","url":"https://addaly.com/learn/building-with-ai/hybrid-search-keywords-still-win","position":37,"of":80,"minutes":9,"answer":"Embeddings blur exact strings and keyword search cannot see paraphrase, so a production search runs both and fuses them by rank rather than score.","keyIdea":"Embeddings blur exact strings and keyword search cannot see paraphrase, so a production search runs both and fuses them by rank rather than score, because BM25 and cosine scores live on scales that cannot be added."},"body":{"format":"markdown","text":"## The queries an embedding fails\n\nOpen the search logs of any real product and a large share of the queries are not questions. They are `error E-4471`, `INV-2024-00931`, `Priya Raghunathan`, a SKU, a function name, a postcode. The last lesson on embeddings explained why these blur: the model breaks a rare string into subword pieces that carry almost no consistent meaning, so `E-4471` and `E-4417` land in nearly the same place.\n\nKeyword search has the opposite blind spot. It cannot see that \"cancel my plan\" and \"stop the subscription\" mean the same thing. It has been the backbone of search for thirty years precisely because on exact terms it is nearly perfect.\n\nA production system needs both, and the standard name for both is hybrid search.\n\n## BM25 in one paragraph\n\nBM25 scores a document for a query by adding up, for each query term, how often the term appears in the document, scaled down by how common the term is across the whole corpus, and normalised for document length so that long documents do not win by volume. Rare terms count for more, which is exactly why it finds an invoice number: that string occurs in one document and nowhere else. Two parameters, `k1` and `b`, control the saturation of term frequency and the strength of length normalisation, and the defaults are fine.\n\nThe free path is everywhere. Postgres has full-text search built in through `tsvector` and `ts_rank`. SQLite has FTS5. Elasticsearch and OpenSearch are free to run. For a corpus that fits in memory, the Python package `rank_bm25` is a few lines:\n\n```python\nfrom rank_bm25 import BM25Okapi\nbm25 = BM25Okapi([doc.split() for doc in docs])\nscores = bm25.get_scores(query.split())\n```\n\n## Fusing two ranked lists\n\nYou now have two lists for one query, one from the vector index and one from BM25, and the wrong way to combine them is to add the scores. A BM25 score is unbounded and depends on the corpus; a cosine score sits between zero and one. Adding them lets whichever has the larger range decide everything.\n\nReciprocal rank fusion ignores the scores and uses only the positions. A document's fused score is the sum, over every list it appears in, of one divided by a constant plus its rank in that list. The constant is conventionally 60, and it exists so that the difference between first and second place is not overwhelming.\n\n```python\ndef rrf(*ranked_lists, k=60):\n    fused = {}\n    for lst in ranked_lists:\n        for rank, doc_id in enumerate(lst, start=1):\n            fused[doc_id] = fused.get(doc_id, 0) + 1 / (k + rank)\n    return sorted(fused, key=fused.get, reverse=True)\n```\n\nA document that is fifth in both lists outranks one that is first in one list and absent from the other, which is usually what you want: agreement between two different signals is strong evidence.\n\n## The gain you should expect\n\nIn published comparisons across mixed corpora, and in most teams' own tests, hybrid retrieval beats either method alone by somewhere between five and fifteen points of recall at ten. The gain is small on corpora of plain-language questions and answers, where the embedding was already doing well, and large on anything full of identifiers: code, logs, catalogues, legal references, medical codes. If your queries include even one identifier in ten, hybrid pays for itself on that tenth alone.\n\n## Tokenisation is where hybrid breaks\n\nBM25 matches tokens, and a token is whatever your analyser decided it is. Three failures to check.\n\nAn English stemmer applied to Hindi does nothing useful, and the default `english` configuration in Postgres will also drop words it considers stop words. For Indian languages use the `simple` configuration or a language-specific analyser, and test with real queries.\n\nCode needs splitting on `camelCase` and `snake_case`; otherwise `getUserById` is one token that matches only itself.\n\nIdentifiers with punctuation get split unpredictably: `INV-2024-00931` may become three tokens, of which `2024` matches ten thousand documents. Index a normalised copy of identifier-like strings with the punctuation removed, so the whole string is one rare token.\n\n## Weighting one side\n\nWhen a query looks like an identifier (a regular expression for digits, uppercase codes and hyphens does the job) the keyword list deserves more weight. RRF accepts a weight per list; giving BM25 double weight on such queries, and the vector list double weight on long natural-language questions, is a cheap improvement that a small classifier can later refine.\n\n## The honest limits\n\nHybrid search does not fix a corpus that lacks the answer, and it does not fix chunks that were cut mid-sentence. It also doubles the index maintenance: two indexes that must be updated together, and a delete that reaches one but not the other leaves a ghost document that keeps appearing in half the results. Put both updates in one transaction where the storage allows it, and where it does not, run a nightly reconciliation that compares the two document sets.\n\n> Figure: Recall over 200 real queries from a support log. Bars (% of queries whose answering chunk is in the top 10): BM25 keyword search alone 64; Embedding search alone 71; Both, fused by reciprocal rank 82. The average hides where the gain comes from. On the 38 queries containing an identifier, BM25 alone reached 88 and embeddings alone 41; on plain-language questions the order reverses. Fusing by rank rather than by score is what lets one list rescue the other."},"check":{"question":"A team combines vector search and BM25 by adding each document's cosine similarity to its BM25 score and sorting. Results are dominated by keyword matches even for paraphrased questions. Why?","options":[{"text":"BM25 is inherently a stronger signal than embeddings and should be expected to dominate.","correct":false,"tempting":"Reads the symptom as a property of the method; BM25 is not stronger, its numbers are merely bigger."},{"text":"The embedding model was not given the query prefix, so its scores are compressed.","correct":false,"tempting":"A real mistake elsewhere, but it would lower vector quality, not make BM25 win by scale."},{"text":"Cosine similarity should be squared before adding, so that its scale matches BM25 and the two signals contribute equally to the combined sum.","correct":false,"tempting":"Invents an ad hoc rescaling; there is no fixed transformation that aligns an unbounded corpus-dependent score with a bounded one."},{"text":"BM25 scores are unbounded and cosine is bounded by one, so the sum is decided by the larger range; fuse by rank instead.","correct":true}]},"prev":{"slug":"vector-search-without-a-vector-database","title":"Vector search: when NumPy is enough, and what an index trades away","url":"https://addaly.com/learn/building-with-ai/vector-search-without-a-vector-database"},"next":{"slug":"reranking-and-how-many-chunks","title":"Reranking, and how many chunks to actually send","url":"https://addaly.com/learn/building-with-ai/reranking-and-how-many-chunks"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Hybrid search: why keyword search is still half the answer', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/hybrid-search-keywords-still-win (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Hybrid search: why keyword search is still half the answer","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"Embeddings blur exact strings and keyword search cannot see paraphrase, so a production search runs both and fuses them by rank rather than score, because BM25 and cosine scores live on scales that cannot be added.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of queries whose answering chunk is in the top 10","title":"Recall over 200 real queries from a support log","series":[{"label":"BM25 keyword search alone","value":64},{"label":"Embedding search alone","value":71},{"label":"Both, fused by reciprocal rank","value":82,"highlight":true}],"caption":"The average hides where the gain comes from. On the 38 queries containing an identifier, BM25 alone reached 88 and embeddings alone 41; on plain-language questions the order reverses. Fusing by rank rather than by score is what lets one list rescue the other."},"alt":"Figure: Recall over 200 real queries from a support log. Bars (% of queries whose answering chunk is in the top 10): BM25 keyword search alone 64; Embedding search alone 71; Both, fused by reciprocal rank 82. The average hides where the gain comes from. On the 38 queries containing an identifier, BM25 alone reached 88 and embeddings alone 41; on plain-language questions the order reverses. Fusing by rank rather than by score is what lets one list rescue the other."}},"blocks":[{"t":"h2","id":"the-queries-an-embedding-fails","runs":[{"text":"The queries an embedding fails"}]},{"t":"p","runs":[{"text":"Open the search logs of any real product and a large share of the queries are not questions. They are "},{"text":"error E-4471","code":true},{"text":", "},{"text":"INV-2024-00931","code":true},{"text":", "},{"text":"Priya Raghunathan","code":true},{"text":", a SKU, a function name, a postcode. The last lesson on embeddings explained why these blur: the model breaks a rare string into subword pieces that carry almost no consistent meaning, so "},{"text":"E-4471","code":true},{"text":" and "},{"text":"E-4417","code":true},{"text":" land in nearly the same place."}]},{"t":"p","runs":[{"text":"Keyword search has the opposite blind spot. It cannot see that “cancel my plan” and “stop the subscription” mean the same thing. It has been the backbone of search for thirty years precisely because on exact terms it is nearly perfect."}]},{"t":"p","runs":[{"text":"A production system needs both, and the standard name for both is hybrid search."}]},{"t":"h2","id":"bm25-in-one-paragraph","runs":[{"text":"BM25 in one paragraph"}]},{"t":"p","runs":[{"text":"BM25 scores a document for a query by adding up, for each query term, how often the term appears in the document, scaled down by how common the term is across the whole corpus, and normalised for document length so that long documents do not win by volume. Rare terms count for more, which is exactly why it finds an invoice number: that string occurs in one document and nowhere else. Two parameters, "},{"text":"k1","code":true},{"text":" and "},{"text":"b","code":true},{"text":", control the saturation of term frequency and the strength of length normalisation, and the defaults are fine."}]},{"t":"p","runs":[{"text":"The free path is everywhere. Postgres has full-text search built in through "},{"text":"tsvector","code":true},{"text":" and "},{"text":"ts_rank","code":true},{"text":". SQLite has FTS5. Elasticsearch and OpenSearch are free to run. For a corpus that fits in memory, the Python package "},{"text":"rank_bm25","code":true},{"text":" is a few lines:"}]},{"t":"code","lang":"python","label":"Python","text":"from rank_bm25 import BM25Okapi\nbm25 = BM25Okapi([doc.split() for doc in docs])\nscores = bm25.get_scores(query.split())"},{"t":"h2","id":"fusing-two-ranked-lists","runs":[{"text":"Fusing two ranked lists"}]},{"t":"p","runs":[{"text":"You now have two lists for one query, one from the vector index and one from BM25, and the wrong way to combine them is to add the scores. A BM25 score is unbounded and depends on the corpus; a cosine score sits between zero and one. Adding them lets whichever has the larger range decide everything."}]},{"t":"p","runs":[{"text":"Reciprocal rank fusion ignores the scores and uses only the positions. A document’s fused score is the sum, over every list it appears in, of one divided by a constant plus its rank in that list. The constant is conventionally 60, and it exists so that the difference between first and second place is not overwhelming."}]},{"t":"code","lang":"python","label":"Python","text":"def rrf(*ranked_lists, k=60):\n    fused = {}\n    for lst in ranked_lists:\n        for rank, doc_id in enumerate(lst, start=1):\n            fused[doc_id] = fused.get(doc_id, 0) + 1 / (k + rank)\n    return sorted(fused, key=fused.get, reverse=True)"},{"t":"p","runs":[{"text":"A document that is fifth in both lists outranks one that is first in one list and absent from the other, which is usually what you want: agreement between two different signals is strong evidence."}]},{"t":"h2","id":"the-gain-you-should-expect","runs":[{"text":"The gain you should expect"}]},{"t":"p","runs":[{"text":"In published comparisons across mixed corpora, and in most teams’ own tests, hybrid retrieval beats either method alone by somewhere between five and fifteen points of recall at ten. The gain is small on corpora of plain-language questions and answers, where the embedding was already doing well, and large on anything full of identifiers: code, logs, catalogues, legal references, medical codes. If your queries include even one identifier in ten, hybrid pays for itself on that tenth alone."}]},{"t":"h2","id":"tokenisation-is-where-hybrid-breaks","runs":[{"text":"Tokenisation is where hybrid breaks"}]},{"t":"p","runs":[{"text":"BM25 matches tokens, and a token is whatever your analyser decided it is. Three failures to check."}]},{"t":"p","runs":[{"text":"An English stemmer applied to Hindi does nothing useful, and the default "},{"text":"english","code":true},{"text":" configuration in Postgres will also drop words it considers stop words. For Indian languages use the "},{"text":"simple","code":true},{"text":" configuration or a language-specific analyser, and test with real queries."}]},{"t":"p","runs":[{"text":"Code needs splitting on "},{"text":"camelCase","code":true},{"text":" and "},{"text":"snake_case","code":true},{"text":"; otherwise "},{"text":"getUserById","code":true},{"text":" is one token that matches only itself."}]},{"t":"p","runs":[{"text":"Identifiers with punctuation get split unpredictably: "},{"text":"INV-2024-00931","code":true},{"text":" may become three tokens, of which "},{"text":"2024","code":true},{"text":" matches ten thousand documents. Index a normalised copy of identifier-like strings with the punctuation removed, so the whole string is one rare token."}]},{"t":"h2","id":"weighting-one-side","runs":[{"text":"Weighting one side"}]},{"t":"p","runs":[{"text":"When a query looks like an identifier (a regular expression for digits, uppercase codes and hyphens does the job) the keyword list deserves more weight. RRF accepts a weight per list; giving BM25 double weight on such queries, and the vector list double weight on long natural-language questions, is a cheap improvement that a small classifier can later refine."}]},{"t":"h2","id":"the-honest-limits","runs":[{"text":"The honest limits"}],"role":"limit"},{"t":"p","runs":[{"text":"Hybrid search does not fix a corpus that lacks the answer, and it does not fix chunks that were cut mid-sentence. It also doubles the index maintenance: two indexes that must be updated together, and a delete that reaches one but not the other leaves a ghost document that keeps appearing in half the results. Put both updates in one transaction where the storage allows it, and where it does not, run a nightly reconciliation that compares the two document sets."}]}],"check":{"question":[{"text":"A team combines vector search and BM25 by adding each document’s cosine similarity to its BM25 score and sorting. Results are dominated by keyword matches even for paraphrased questions. Why?"}],"options":[{"text":[{"text":"BM25 is inherently a stronger signal than embeddings and should be expected to dominate."}],"correct":false,"tempting":[{"text":"Reads the symptom as a property of the method; BM25 is not stronger, its numbers are merely bigger."}]},{"text":[{"text":"The embedding model was not given the query prefix, so its scores are compressed."}],"correct":false,"tempting":[{"text":"A real mistake elsewhere, but it would lower vector quality, not make BM25 win by scale."}]},{"text":[{"text":"Cosine similarity should be squared before adding, so that its scale matches BM25 and the two signals contribute equally to the combined sum."}],"correct":false,"tempting":[{"text":"Invents an ad hoc rescaling; there is no fixed transformation that aligns an unbounded corpus-dependent score with a bounded one."}]},{"text":[{"text":"BM25 scores are unbounded and cosine is bounded by one, so the sum is decided by the larger range; fuse by rank instead."}],"correct":true,"tempting":null}],"order":[2,3,1,0],"why":null},"prev":{"slug":"vector-search-without-a-vector-database","title":"Vector search: when NumPy is enough, and what an index trades away","minutes":9},"next":{"slug":"reranking-and-how-many-chunks","title":"Reranking, and how many chunks to actually send","minutes":9}}}