{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"the-query-is-not-the-question","title":"Rewriting the query before you search","url":"https://addaly.com/learn/context-engineering/the-query-is-not-the-question","position":24,"of":87,"minutes":9,"answer":"The user's message is rarely a good search query: resolve references from the conversation, split compound questions, expand paraphrases and optionally search with a hypothetical answer.","keyIdea":"The user's message is rarely a good search query: resolve references from the conversation, split compound questions, expand paraphrases and optionally search with a hypothetical answer, but always fuse the original query back in so a bad rewrite cannot silently replace the real one."},"body":{"format":"markdown","text":"## The string you search with is a design decision\n\nThe user typed \"what about the second one?\". You cannot search with that. It contains no content words, and the thing it refers to is three turns above it in a conversation the retriever has never seen.\n\nThis is the most common retrieval failure in conversational products and it is not a retrieval failure at all. The retriever did its job on the string it was given. The string was wrong.\n\nFour transformations sit between a user's message and a good query, and each solves a different problem.\n\n## 1. Resolve the conversation\n\nRewrite the latest message into a standalone question using the last few turns.\n\n```\nGiven this conversation, rewrite the final user message as a single\nself-contained question. Keep the user's own terminology. If it is already\nself-contained, return it unchanged.\n```\n\n\"What about the second one?\" becomes \"What is the refund window for the standard plan?\". Now it can be embedded, keyword-searched and cached.\n\nTwo details that matter. Keep the user's own vocabulary: a rewrite that translates their words into your internal jargon can search a corpus that uses the user's words and lose. And return the original unchanged when it is already standalone, because rewriting a good query is a chance to make it worse for no benefit.\n\n## 2. Decompose the compound question\n\n\"Compare the refund policies for the standard and enterprise plans.\" One embedding of that sits somewhere between the two topics and may retrieve neither well.\n\nSplit into sub-queries, retrieve for each, merge. It costs one extra model call and roughly doubles the retrieval work, and it is the difference between a comparison built on both sources and one built on whichever happened to rank higher.\n\nThe signal for when to do it: questions containing \"and\", \"versus\", \"compare\", \"both\", or two named entities. A cheap classifier or a short prompt catches most of them, and you can skip decomposition on the rest.\n\n## 3. Expand and fuse\n\nGenerate three or four paraphrases of the query, retrieve for all of them, and fuse the ranked lists. This raises recall because different phrasings match different vocabulary in the corpus, and it does it without needing the corpus to be re-indexed.\n\nThe fusion is the same reciprocal-rank method the hybrid search lesson uses next. The cost is n times the retrieval work, which is usually cheap compared with the generation call.\n\n## 4. HyDE: search with a hypothetical answer\n\nAsk the model to write the answer it *thinks* is correct, then embed that and search with it.\n\nThe reasoning is that a question and its answer are not lexically or distributionally alike: a question is short, interrogative and vague; an answer is declarative and uses the domain's vocabulary. Searching with a fabricated answer moves the query vector into the region where real answers live.\n\nIt works, sometimes strikingly, on domains where the model has genuine background knowledge. It fails on private material the model knows nothing about: the hypothetical answer is then a confident invention that pulls retrieval towards documents about a thing that does not exist in your corpus. Always fuse HyDE's results with the original query's rather than replacing them, so a bad hypothesis costs you nothing.\n\n## The cost you are adding\n\nEvery one of these is a model call before retrieval, which means latency in front of the user's first token. A small fast model does rewriting well; this is not a task that needs your best one.\n\nBudget honestly: a rewrite call at 300 milliseconds plus retrieval at 80 milliseconds plus generation is a noticeably slower product than retrieval plus generation. Three mitigations, in order of usefulness:\n\n- **Skip it when it is not needed.** First message of a conversation, no pronouns, more than four content words: search directly. Most queries need no rewriting.\n- **Cache rewrites** keyed on the conversation tail. Repeated and near-repeated questions are common.\n- **Run rewriting in parallel with a direct search** of the raw query, and fuse. The direct result is free and often good.\n\n## Keep the original in the mix\n\nThe rule that prevents the whole class of rewriting disasters: **never discard the user's original query.** Rewriting adds a hypothesis; it should not replace the evidence.\n\nA rewrite can invent a constraint that was not there (\"for orders placed in India\"), narrow a broad question, or resolve a pronoun to the wrong antecedent. All three produce a confidently wrong retrieval set. Fusing the original query's results in costs one more search and bounds the damage.\n\n## What to log\n\nLog the original message, the rewritten query, and which of them produced each retrieved passage. When somebody reports a bad answer, this one line separates three otherwise indistinguishable failures: a bad rewrite, a good rewrite that retrieved badly, and a corpus that does not contain the answer.\n\nWithout it, the rewritten query is invisible, and you will spend the investigation looking at a question the retriever never saw."},"check":{"question":"A team adds HyDE to a corpus of internal engineering documents about their own proprietary systems, and retrieval quality falls. What is the mechanism?","options":[{"text":"HyDE embeddings are longer, so they exceed the embedding model's input limit and get truncated.","correct":false,"tempting":"Truncation is a real hazard for long inputs, but a short hypothetical answer sits well inside typical embedding limits."},{"text":"The model has no knowledge of the proprietary systems, so the hypothetical answer is an invention that moves the query vector away from the real documents.","correct":true},{"text":"Hypothetical answers are declarative, and declarative text matches documentation less well than interrogative text does.","correct":false,"tempting":"Inverts the premise of the technique; matching declarative answers to declarative documents is exactly why it usually helps."},{"text":"HyDE requires a reranker to work, and without one the fused list is unordered.","correct":false,"tempting":"Invents a dependency; HyDE produces a ranked list on its own, and reranking is an independent later stage."}]},"prev":{"slug":"contextual-chunk-augmentation","title":"Giving each chunk its own context","url":"https://addaly.com/learn/context-engineering/contextual-chunk-augmentation"},"next":{"slug":"keywords-and-vectors-together","title":"Keywords still win, so run both","url":"https://addaly.com/learn/context-engineering/keywords-and-vectors-together"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rewriting the query before you search', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/the-query-is-not-the-question (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rewriting the query before you search","courseTitle":"Context Engineering","module":{"index":3,"slug":"retrieval-as-a-pipeline","title":"Retrieval as a context pipeline"},"answer":null,"keyIdea":"The user’s message is rarely a good search query: resolve references from the conversation, split compound questions, expand paraphrases and optionally search with a hypothetical answer, but always fuse the original query back in so a bad rewrite cannot silently replace the real one.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-string-you-search-with-is-a-design-decision","runs":[{"text":"The string you search with is a design decision"}]},{"t":"p","runs":[{"text":"The user typed “what about the second one?”. You cannot search with that. It contains no content words, and the thing it refers to is three turns above it in a conversation the retriever has never seen."}]},{"t":"p","runs":[{"text":"This is the most common retrieval failure in conversational products and it is not a retrieval failure at all. The retriever did its job on the string it was given. The string was wrong."}]},{"t":"p","runs":[{"text":"Four transformations sit between a user’s message and a good query, and each solves a different problem."}]},{"t":"h2","id":"1-resolve-the-conversation","runs":[{"text":"1. Resolve the conversation"}]},{"t":"p","runs":[{"text":"Rewrite the latest message into a standalone question using the last few turns."}]},{"t":"code","lang":"","label":"Text","text":"Given this conversation, rewrite the final user message as a single\nself-contained question. Keep the user's own terminology. If it is already\nself-contained, return it unchanged."},{"t":"p","runs":[{"text":"“What about the second one?” becomes “What is the refund window for the standard plan?”. Now it can be embedded, keyword-searched and cached."}]},{"t":"p","runs":[{"text":"Two details that matter. Keep the user’s own vocabulary: a rewrite that translates their words into your internal jargon can search a corpus that uses the user’s words and lose. And return the original unchanged when it is already standalone, because rewriting a good query is a chance to make it worse for no benefit."}]},{"t":"h2","id":"2-decompose-the-compound-question","runs":[{"text":"2. Decompose the compound question"}]},{"t":"p","runs":[{"text":"“Compare the refund policies for the standard and enterprise plans.” One embedding of that sits somewhere between the two topics and may retrieve neither well."}]},{"t":"p","runs":[{"text":"Split into sub-queries, retrieve for each, merge. It costs one extra model call and roughly doubles the retrieval work, and it is the difference between a comparison built on both sources and one built on whichever happened to rank higher."}]},{"t":"p","runs":[{"text":"The signal for when to do it: questions containing “and”, “versus”, “compare”, “both”, or two named entities. A cheap classifier or a short prompt catches most of them, and you can skip decomposition on the rest."}]},{"t":"h2","id":"3-expand-and-fuse","runs":[{"text":"3. Expand and fuse"}]},{"t":"p","runs":[{"text":"Generate three or four paraphrases of the query, retrieve for all of them, and fuse the ranked lists. This raises recall because different phrasings match different vocabulary in the corpus, and it does it without needing the corpus to be re-indexed."}]},{"t":"p","runs":[{"text":"The fusion is the same reciprocal-rank method the hybrid search lesson uses next. The cost is n times the retrieval work, which is usually cheap compared with the generation call."}]},{"t":"h2","id":"4-hyde-search-with-a-hypothetical-answer","runs":[{"text":"4. HyDE: search with a hypothetical answer"}]},{"t":"p","runs":[{"text":"Ask the model to write the answer it "},{"text":"thinks","i":true},{"text":" is correct, then embed that and search with it."}]},{"t":"p","runs":[{"text":"The reasoning is that a question and its answer are not lexically or distributionally alike: a question is short, interrogative and vague; an answer is declarative and uses the domain’s vocabulary. Searching with a fabricated answer moves the query vector into the region where real answers live."}]},{"t":"p","runs":[{"text":"It works, sometimes strikingly, on domains where the model has genuine background knowledge. It fails on private material the model knows nothing about: the hypothetical answer is then a confident invention that pulls retrieval towards documents about a thing that does not exist in your corpus. Always fuse HyDE’s results with the original query’s rather than replacing them, so a bad hypothesis costs you nothing."}]},{"t":"h2","id":"the-cost-you-are-adding","runs":[{"text":"The cost you are adding"}]},{"t":"p","runs":[{"text":"Every one of these is a model call before retrieval, which means latency in front of the user’s first token. A small fast model does rewriting well; this is not a task that needs your best one."}]},{"t":"p","runs":[{"text":"Budget honestly: a rewrite call at 300 milliseconds plus retrieval at 80 milliseconds plus generation is a noticeably slower product than retrieval plus generation. Three mitigations, in order of usefulness:"}]},{"t":"ul","items":[[{"text":"Skip it when it is not needed.","b":true},{"text":" First message of a conversation, no pronouns, more than four content words: search directly. Most queries need no rewriting."}],[{"text":"Cache rewrites","b":true},{"text":" keyed on the conversation tail. Repeated and near-repeated questions are common."}],[{"text":"Run rewriting in parallel with a direct search","b":true},{"text":" of the raw query, and fuse. The direct result is free and often good."}]]},{"t":"h2","id":"keep-the-original-in-the-mix","runs":[{"text":"Keep the original in the mix"}]},{"t":"p","runs":[{"text":"The rule that prevents the whole class of rewriting disasters: "},{"text":"never discard the user’s original query.","b":true},{"text":" Rewriting adds a hypothesis; it should not replace the evidence."}]},{"t":"p","runs":[{"text":"A rewrite can invent a constraint that was not there (“for orders placed in India”), narrow a broad question, or resolve a pronoun to the wrong antecedent. All three produce a confidently wrong retrieval set. Fusing the original query’s results in costs one more search and bounds the damage."}]},{"t":"h2","id":"what-to-log","runs":[{"text":"What to log"}]},{"t":"p","runs":[{"text":"Log the original message, the rewritten query, and which of them produced each retrieved passage. When somebody reports a bad answer, this one line separates three otherwise indistinguishable failures: a bad rewrite, a good rewrite that retrieved badly, and a corpus that does not contain the answer."}]},{"t":"p","runs":[{"text":"Without it, the rewritten query is invisible, and you will spend the investigation looking at a question the retriever never saw."}]}],"check":{"question":[{"text":"A team adds HyDE to a corpus of internal engineering documents about their own proprietary systems, and retrieval quality falls. What is the mechanism?"}],"options":[{"text":[{"text":"HyDE embeddings are longer, so they exceed the embedding model’s input limit and get truncated."}],"correct":false,"tempting":[{"text":"Truncation is a real hazard for long inputs, but a short hypothetical answer sits well inside typical embedding limits."}]},{"text":[{"text":"The model has no knowledge of the proprietary systems, so the hypothetical answer is an invention that moves the query vector away from the real documents."}],"correct":true,"tempting":null},{"text":[{"text":"Hypothetical answers are declarative, and declarative text matches documentation less well than interrogative text does."}],"correct":false,"tempting":[{"text":"Inverts the premise of the technique; matching declarative answers to declarative documents is exactly why it usually helps."}]},{"text":[{"text":"HyDE requires a reranker to work, and without one the fused list is unordered."}],"correct":false,"tempting":[{"text":"Invents a dependency; HyDE produces a ranked list on its own, and reranking is an independent later stage."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"contextual-chunk-augmentation","title":"Giving each chunk its own context","minutes":9},"next":{"slug":"keywords-and-vectors-together","title":"Keywords still win, so run both","minutes":9}}}