{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"keywords-and-vectors-together","title":"Keywords still win, so run both","url":"https://addaly.com/learn/context-engineering/keywords-and-vectors-together","position":25,"of":87,"minutes":9,"answer":"Dense and sparse retrieval fail in opposite directions, so run both and fuse by reciprocal rank rather than by normalising incomparable scores, and check that your keyword analyser actually tokenises the languages you serve.","keyIdea":"Dense and sparse retrieval fail in opposite directions, so run both and fuse by reciprocal rank rather than by normalising incomparable scores, and check that your keyword analyser actually tokenises the languages you serve, because that failure is completely silent."},"body":{"format":"markdown","text":"## What each method is good at\n\nDense vector search finds text that means something similar. Sparse keyword search finds text that contains the same words. They fail in complementary ways, which is why hybrid retrieval is not a compromise but the default.\n\nKeyword search (BM25 in practice) wins decisively on:\n\n- **Identifiers.** `ORD-88213`, `ECONNREFUSED`, `Section 80C`, a part number, a person's surname. These are rare tokens; rarity is exactly what BM25 rewards and exactly what a dense embedding smooths away.\n- **Exact phrases** the user is quoting from something they are looking at.\n- **Domain jargon the embedding model never saw.** Product names, internal acronyms, terms coined after the embedding model was trained.\n- **Negation and precise wording**, indirectly: the words are matched literally, so \"not\" is a token like any other rather than a nuance the embedding blurs.\n\nDense search wins on:\n\n- **Vocabulary mismatch.** \"My money hasn't come back\" against \"reimbursement processing timelines\".\n- **Conceptual questions** with no distinctive shared terms.\n- **Cross-lingual retrieval**, with a multilingual embedding model: a Hindi question can retrieve an English document, which BM25 cannot do at all.\n\nA system with only one of them has a predictable hole, and users find it in the first week.\n\n## BM25 in one paragraph\n\nBM25 scores a document by, for each query term, how often it appears in the document, damped so the tenth occurrence adds little, multiplied by how rare the term is across the corpus, and divided down if the document is long relative to average. That is the entire idea. Its strength is the rarity weighting: a match on `ECONNREFUSED` counts for enormously more than a match on `error`, without anybody configuring that.\n\nFree implementations everywhere. `rank_bm25` in Python for small corpora, SQLite's FTS5 for anything up to a few million documents on a single machine, PostgreSQL's `tsvector` if your data already lives there, OpenSearch or Elasticsearch at scale. None of this needs a GPU or a hosted service.\n\n## Fusing the two lists\n\nYou have two ranked lists with incomparable scores: a cosine of 0.71 and a BM25 of 14.3. Do not try to add them. Score normalisation across methods is fragile: it depends on the query, the corpus and the day.\n\nReciprocal rank fusion ignores the scores and uses only the positions:\n\n```python\ndef rrf(lists, k=60):\n    scores = defaultdict(float)\n    for lst in lists:\n        for rank, doc_id in enumerate(lst, start=1):\n            scores[doc_id] += 1.0 / (k + rank)\n    return sorted(scores, key=scores.get, reverse=True)\n```\n\nThe constant `k=60` is the value from the original paper and it works well enough that almost nobody changes it. Its job is to flatten the difference between the top few positions so that one list cannot dominate on rank alone; larger values flatten more.\n\nRRF is robust, needs no tuning, and composes with any number of lists: vector, keyword, one per rewritten query, one per sub-question. Its limitation is the flip side of its strength: it throws away magnitude, so a query where one method found an obviously perfect match gets no extra credit for that. Where you have a reranker downstream this does not matter much, because the reranker restores a meaningful score over the fused candidates.\n\n## Getting the tokenisation right for your languages\n\nBM25 needs words, and word boundaries are a language-specific problem that default configurations quietly get wrong.\n\nEnglish stemming is standard and helps. Hindi, Bengali and Tamil are heavily inflected and need their own analysers or the same word in two forms will not match. Chinese, Japanese and Thai have no spaces at all and require a segmenter: without one, a whitespace tokeniser produces one enormous token per sentence and keyword search silently returns nothing.\n\nCheck this explicitly if you serve those languages. The failure is silent: no error, no warning, just a keyword index that never contributes to the fused results, and a hybrid system that is a dense-only system with extra latency.\n\n## Filters apply to both\n\nEverything the earlier lesson said about pre-filtering applies to each retriever separately. A permission filter applied to only one arm of a hybrid search leaks through the other, and it is an easy mistake because the two arms are usually different pieces of infrastructure with different filter syntax.\n\nWrite one function that builds the filter and have both retrievers consume it, so the two cannot drift apart.\n\n## What to expect from adding it\n\nAdding BM25 to a dense-only system typically produces a solid recall improvement, concentrated entirely in a specific slice of queries: the ones containing identifiers, names and rare terms. On conceptual questions it changes almost nothing.\n\n> Figure: Adding keyword search, split by kind of question. Bars (%): Identifier lookups, dense only 41; Identifier lookups, fused 88; Conceptual questions, dense only 79; Conceptual questions, fused 82. Recall at ten on the same corpus, dense-only against dense fused with BM25 by reciprocal rank. The average across both slices moves about six points, which is why an average is the wrong way to report this: one category of query went from failing outright to working, and those were the queries users were complaining about.\n\nWhich means the headline average understates it. Report it split by query type, because \"3 points of recall\" hides the fact that a category of queries went from failing outright to working, and those are usually the queries where a user was staring at an error code and knew the answer existed."},"check":{"question":"A hybrid system serving Hindi queries shows the keyword arm contributing almost no results, while the vector arm works. Latency is unchanged. What should you check first?","options":[{"text":"Whether the permission filter is being applied to both arms with the same syntax.","correct":false,"tempting":"A genuine source of divergence between arms, but it would exclude documents rather than prevent matching entirely."},{"text":"Whether the RRF constant is too high, flattening the keyword arm's contribution.","correct":false,"tempting":"The constant flattens differences between ranks within a list; it cannot suppress one list relative to another."},{"text":"Whether the analyser handles Hindi morphology, since a default English analyser will fail to match inflected forms.","correct":true},{"text":"Whether the embedding model is multilingual, since a monolingual one would distort the fused ordering.","correct":false,"tempting":"Would explain a failing vector arm, but the vector arm is the one that works."}]},"prev":{"slug":"the-query-is-not-the-question","title":"Rewriting the query before you search","url":"https://addaly.com/learn/context-engineering/the-query-is-not-the-question"},"next":{"slug":"reranking-to-fit-the-budget","title":"Reranking, and why it reads better than the retriever","url":"https://addaly.com/learn/context-engineering/reranking-to-fit-the-budget"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Keywords still win, so run both', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/keywords-and-vectors-together (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Keywords still win, so run both","courseTitle":"Context Engineering","module":{"index":3,"slug":"retrieval-as-a-pipeline","title":"Retrieval as a context pipeline"},"answer":null,"keyIdea":"Dense and sparse retrieval fail in opposite directions, so run both and fuse by reciprocal rank rather than by normalising incomparable scores, and check that your keyword analyser actually tokenises the languages you serve, because that failure is completely silent.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"%","title":"Adding keyword search, split by kind of question","series":[{"label":"Identifier lookups, dense only","value":41},{"label":"Identifier lookups, fused","value":88,"highlight":true},{"label":"Conceptual questions, dense only","value":79},{"label":"Conceptual questions, fused","value":82}],"caption":"Recall at ten on the same corpus, dense-only against dense fused with BM25 by reciprocal rank. The average across both slices moves about six points, which is why an average is the wrong way to report this: one category of query went from failing outright to working, and those were the queries users were complaining about."},"alt":"Figure: Adding keyword search, split by kind of question. Bars (%): Identifier lookups, dense only 41; Identifier lookups, fused 88; Conceptual questions, dense only 79; Conceptual questions, fused 82. Recall at ten on the same corpus, dense-only against dense fused with BM25 by reciprocal rank. The average across both slices moves about six points, which is why an average is the wrong way to report this: one category of query went from failing outright to working, and those were the queries users were complaining about."}},"blocks":[{"t":"h2","id":"what-each-method-is-good-at","runs":[{"text":"What each method is good at"}]},{"t":"p","runs":[{"text":"Dense vector search finds text that means something similar. Sparse keyword search finds text that contains the same words. They fail in complementary ways, which is why hybrid retrieval is not a compromise but the default."}]},{"t":"p","runs":[{"text":"Keyword search (BM25 in practice) wins decisively on:"}]},{"t":"ul","items":[[{"text":"Identifiers.","b":true},{"text":" "},{"text":"ORD-88213","code":true},{"text":", "},{"text":"ECONNREFUSED","code":true},{"text":", "},{"text":"Section 80C","code":true},{"text":", a part number, a person’s surname. These are rare tokens; rarity is exactly what BM25 rewards and exactly what a dense embedding smooths away."}],[{"text":"Exact phrases","b":true},{"text":" the user is quoting from something they are looking at."}],[{"text":"Domain jargon the embedding model never saw.","b":true},{"text":" Product names, internal acronyms, terms coined after the embedding model was trained."}],[{"text":"Negation and precise wording","b":true},{"text":", indirectly: the words are matched literally, so “not” is a token like any other rather than a nuance the embedding blurs."}]]},{"t":"p","runs":[{"text":"Dense search wins on:"}]},{"t":"ul","items":[[{"text":"Vocabulary mismatch.","b":true},{"text":" “My money hasn’t come back” against “reimbursement processing timelines”."}],[{"text":"Conceptual questions","b":true},{"text":" with no distinctive shared terms."}],[{"text":"Cross-lingual retrieval","b":true},{"text":", with a multilingual embedding model: a Hindi question can retrieve an English document, which BM25 cannot do at all."}]]},{"t":"p","runs":[{"text":"A system with only one of them has a predictable hole, and users find it in the first week."}]},{"t":"h2","id":"bm25-in-one-paragraph","runs":[{"text":"BM25 in one paragraph"}]},{"t":"p","runs":[{"text":"BM25 scores a document by, for each query term, how often it appears in the document, damped so the tenth occurrence adds little, multiplied by how rare the term is across the corpus, and divided down if the document is long relative to average. That is the entire idea. Its strength is the rarity weighting: a match on "},{"text":"ECONNREFUSED","code":true},{"text":" counts for enormously more than a match on "},{"text":"error","code":true},{"text":", without anybody configuring that."}]},{"t":"p","runs":[{"text":"Free implementations everywhere. "},{"text":"rank_bm25","code":true},{"text":" in Python for small corpora, SQLite’s FTS5 for anything up to a few million documents on a single machine, PostgreSQL’s "},{"text":"tsvector","code":true},{"text":" if your data already lives there, OpenSearch or Elasticsearch at scale. None of this needs a GPU or a hosted service."}]},{"t":"h2","id":"fusing-the-two-lists","runs":[{"text":"Fusing the two lists"}]},{"t":"p","runs":[{"text":"You have two ranked lists with incomparable scores: a cosine of 0.71 and a BM25 of 14.3. Do not try to add them. Score normalisation across methods is fragile: it depends on the query, the corpus and the day."}]},{"t":"p","runs":[{"text":"Reciprocal rank fusion ignores the scores and uses only the positions:"}]},{"t":"code","lang":"python","label":"Python","text":"def rrf(lists, k=60):\n    scores = defaultdict(float)\n    for lst in lists:\n        for rank, doc_id in enumerate(lst, start=1):\n            scores[doc_id] += 1.0 / (k + rank)\n    return sorted(scores, key=scores.get, reverse=True)"},{"t":"p","runs":[{"text":"The constant "},{"text":"k=60","code":true},{"text":" is the value from the original paper and it works well enough that almost nobody changes it. Its job is to flatten the difference between the top few positions so that one list cannot dominate on rank alone; larger values flatten more."}]},{"t":"p","runs":[{"text":"RRF is robust, needs no tuning, and composes with any number of lists: vector, keyword, one per rewritten query, one per sub-question. Its limitation is the flip side of its strength: it throws away magnitude, so a query where one method found an obviously perfect match gets no extra credit for that. Where you have a reranker downstream this does not matter much, because the reranker restores a meaningful score over the fused candidates."}]},{"t":"h2","id":"getting-the-tokenisation-right-for-your-languages","runs":[{"text":"Getting the tokenisation right for your languages"}]},{"t":"p","runs":[{"text":"BM25 needs words, and word boundaries are a language-specific problem that default configurations quietly get wrong."}]},{"t":"p","runs":[{"text":"English stemming is standard and helps. Hindi, Bengali and Tamil are heavily inflected and need their own analysers or the same word in two forms will not match. Chinese, Japanese and Thai have no spaces at all and require a segmenter: without one, a whitespace tokeniser produces one enormous token per sentence and keyword search silently returns nothing."}]},{"t":"p","runs":[{"text":"Check this explicitly if you serve those languages. The failure is silent: no error, no warning, just a keyword index that never contributes to the fused results, and a hybrid system that is a dense-only system with extra latency."}]},{"t":"h2","id":"filters-apply-to-both","runs":[{"text":"Filters apply to both"}]},{"t":"p","runs":[{"text":"Everything the earlier lesson said about pre-filtering applies to each retriever separately. A permission filter applied to only one arm of a hybrid search leaks through the other, and it is an easy mistake because the two arms are usually different pieces of infrastructure with different filter syntax."}]},{"t":"p","runs":[{"text":"Write one function that builds the filter and have both retrievers consume it, so the two cannot drift apart."}]},{"t":"h2","id":"what-to-expect-from-adding-it","runs":[{"text":"What to expect from adding it"}]},{"t":"p","runs":[{"text":"Adding BM25 to a dense-only system typically produces a solid recall improvement, concentrated entirely in a specific slice of queries: the ones containing identifiers, names and rare terms. On conceptual questions it changes almost nothing."}]},{"t":"p","runs":[{"text":"Which means the headline average understates it. Report it split by query type, because “3 points of recall” hides the fact that a category of queries went from failing outright to working, and those are usually the queries where a user was staring at an error code and knew the answer existed."}]}],"check":{"question":[{"text":"A hybrid system serving Hindi queries shows the keyword arm contributing almost no results, while the vector arm works. Latency is unchanged. What should you check first?"}],"options":[{"text":[{"text":"Whether the permission filter is being applied to both arms with the same syntax."}],"correct":false,"tempting":[{"text":"A genuine source of divergence between arms, but it would exclude documents rather than prevent matching entirely."}]},{"text":[{"text":"Whether the RRF constant is too high, flattening the keyword arm’s contribution."}],"correct":false,"tempting":[{"text":"The constant flattens differences between ranks within a list; it cannot suppress one list relative to another."}]},{"text":[{"text":"Whether the analyser handles Hindi morphology, since a default English analyser will fail to match inflected forms."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the embedding model is multilingual, since a monolingual one would distort the fused ordering."}],"correct":false,"tempting":[{"text":"Would explain a failing vector arm, but the vector arm is the one that works."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"the-query-is-not-the-question","title":"Rewriting the query before you search","minutes":9},"next":{"slug":"reranking-to-fit-the-budget","title":"Reranking, and why it reads better than the retriever","minutes":9}}}