{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"rewriting-the-question","title":"Rewriting the question before you search it","url":"https://addaly.com/learn/building-with-ai/rewriting-the-question","position":39,"of":80,"minutes":9,"answer":"The text a user typed is rarely the best search query, because follow-up turns omit the topic, terse questions omit the vocabulary and mixed-language questions miss the corpus.","keyIdea":"The text a user typed is rarely the best search query, because follow-up turns omit the topic, terse questions omit the vocabulary and mixed-language questions miss the corpus, so a cheap rewriting call before retrieval (condensing, expanding, decomposing or extracting filters) is usually the largest single improvement available."},"body":{"format":"markdown","text":"## The query the user typed is not the query to search\n\nA user asks \"what does the Pro plan cost?\", gets an answer, and then types \"and for enterprise?\". Embed that second message on its own and it lands nowhere useful. The word that matters, cost, is in the previous turn. The retrieval system, which sees only the string it was given, returns chunks about enterprise onboarding, enterprise security, anything with the word in it, and the model answers from the wrong passages.\n\nThis is the most common failure in chat-over-documents, and it has nothing to do with embeddings. The fix is to rewrite the message into a standalone query before searching. A small, fast model, given the last few turns and asked to produce one self-contained search query, returns \"enterprise plan pricing\" in a hundred milliseconds for a fraction of a cent. That one step fixes more follow-up questions than any change to the index.\n\n```python\nREWRITE = \"\"\"Given the conversation and the latest message, write one\nstandalone search query that captures what the user is asking.\nOutput the query only.\"\"\"\n```\n\n## Expansion\n\nTerse queries miss. \"refund\" against a corpus that says \"returns and reimbursement\" may never meet. Ask the model for three paraphrases, search all four, and fuse the lists with reciprocal rank fusion from the hybrid-search lesson. Recall goes up on short queries; the price is three extra searches, which are cheap, and one extra model call, which is not free in latency. Reserve it for queries under a handful of words.\n\n## Decomposition\n\n\"Compare the refund policy for annual and monthly plans\" needs two facts that live in two places. One search with that sentence finds whichever place matches the sentence better and misses the other. Decompose into two sub-queries, search each, and give the model both sets. The structured-output techniques from the earlier module do this cleanly: ask for a list of sub-questions as JSON, cap it at three, and search them in parallel.\n\n## HyDE, and why it sometimes helps\n\nHypothetical document embedding asks the model to write a plausible answer to the question first, then embeds that fabricated answer and searches with it. The mechanism is that passage-to-passage similarity is a better-matched comparison than question-to-passage: a fake answer uses the vocabulary and shape of a real one. Reported gains range from substantial to none, and the reason is visible in the mechanism. When the model knows the domain's vocabulary, its fake answer resembles the real passage. In a niche domain it fabricates vocabulary that appears nowhere, and the search gets worse. It also adds a generation call of several hundred milliseconds. Try it on your evaluation set; keep it only if it moves the number.\n\n## Extracting filters from the question\n\n\"Invoices from March for Acme\" is a metadata filter wearing the clothes of a query. Ask the model for structured output (a date range, a customer name, a document type) alongside the free-text part, and turn the structured half into a filter on the index. A search restricted to one customer's March invoices is nearly always better than a similarity search over everything, and this is where the permission-and-metadata work of the next lesson pays back.\n\n## Spelling, transliteration and language\n\nKeyword search fails on a misspelling completely; embeddings tolerate small ones. A rewrite step that corrects spelling helps the keyword half of a hybrid system. For a user writing in Hindi or Hinglish against English documents, the rewrite can translate the query, which is often the difference between a hit and nothing. Be honest with yourself about the risk: translation can lose a nuance the user cared about, and a rewrite that changes the meaning is worse than no rewrite. Log both versions so you can see when that happened.\n\n## The costs, and a routing rule\n\nEvery rewrite is a model call: roughly 200 to 500 milliseconds and a fraction of a cent with a small model. Stacked (condense, then expand, then decompose) that is a second and a half before the search has started. So route rather than always rewrite.\n\n- First turn of a conversation, and the message is a complete question: search it as typed.\n- Follow-up turn: condense.\n- Under five words: expand.\n- Contains \"and\", \"compare\", \"versus\" or two question marks: consider decomposition.\n- Looks like an identifier: do not rewrite at all; a rewrite will helpfully \"correct\" the invoice number.\n\n## Log the rewritten query\n\nWhen retrieval fails you need to know which stage failed. Was the rewritten query wrong, or was the search wrong given a right query? Log the original message, the rewritten query, every sub-query, and the fused candidate list, keyed by request. A failure you cannot decompose into stages is one you will tune blindly, and the rewrite stage is the one that most often turns out to be the culprit when a user reports \"it answered a different question\"."},"check":{"question":"A chat system over product documentation answers first questions well but answers follow-ups like \"and for enterprise?\" from irrelevant passages. What is the mechanism?","options":[{"text":"The embedding model has not been given the query prefix, so follow-up questions embed poorly.","correct":false,"tempting":"Would degrade every query equally, not specifically the follow-ups."},{"text":"The chunks are too small to contain both the plan name and the price, so the answer is split across chunks and the model cannot join the two halves back together.","correct":false,"tempting":"Chunk size would affect first questions as much as follow-ups; the pattern points at the query, not the corpus."},{"text":"The follow-up is searched as typed, without the topic from the previous turn, so its embedding lands near passages that merely contain \"enterprise\".","correct":true},{"text":"The model is ignoring retrieved context on later turns because the conversation has grown too long.","correct":false,"tempting":"Blames generation for a retrieval failure; the wrong passages were retrieved before the model saw anything."}]},"prev":{"slug":"reranking-and-how-many-chunks","title":"Reranking, and how many chunks to actually send","url":"https://addaly.com/learn/building-with-ai/reranking-and-how-many-chunks"},"next":{"slug":"ingestion-pdfs-tables-and-scans","title":"Ingestion: PDFs, tables and scans, where most retrieval actually fails","url":"https://addaly.com/learn/building-with-ai/ingestion-pdfs-tables-and-scans"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rewriting the question before you search it', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/rewriting-the-question (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rewriting the question before you search it","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"The text a user typed is rarely the best search query, because follow-up turns omit the topic, terse questions omit the vocabulary and mixed-language questions miss the corpus, so a cheap rewriting call before retrieval (condensing, expanding, decomposing or extracting filters) is usually the largest single improvement available.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-query-the-user-typed-is-not-the-query-to-search","runs":[{"text":"The query the user typed is not the query to search"}]},{"t":"p","runs":[{"text":"A user asks “what does the Pro plan cost?”, gets an answer, and then types “and for enterprise?”. Embed that second message on its own and it lands nowhere useful. The word that matters, cost, is in the previous turn. The retrieval system, which sees only the string it was given, returns chunks about enterprise onboarding, enterprise security, anything with the word in it, and the model answers from the wrong passages."}]},{"t":"p","runs":[{"text":"This is the most common failure in chat-over-documents, and it has nothing to do with embeddings. The fix is to rewrite the message into a standalone query before searching. A small, fast model, given the last few turns and asked to produce one self-contained search query, returns “enterprise plan pricing” in a hundred milliseconds for a fraction of a cent. That one step fixes more follow-up questions than any change to the index."}]},{"t":"code","lang":"python","label":"Python","text":"REWRITE = \"\"\"Given the conversation and the latest message, write one\nstandalone search query that captures what the user is asking.\nOutput the query only.\"\"\""},{"t":"h2","id":"expansion","runs":[{"text":"Expansion"}]},{"t":"p","runs":[{"text":"Terse queries miss. “refund” against a corpus that says “returns and reimbursement” may never meet. Ask the model for three paraphrases, search all four, and fuse the lists with reciprocal rank fusion from the hybrid-search lesson. Recall goes up on short queries; the price is three extra searches, which are cheap, and one extra model call, which is not free in latency. Reserve it for queries under a handful of words."}]},{"t":"h2","id":"decomposition","runs":[{"text":"Decomposition"}]},{"t":"p","runs":[{"text":"“Compare the refund policy for annual and monthly plans” needs two facts that live in two places. One search with that sentence finds whichever place matches the sentence better and misses the other. Decompose into two sub-queries, search each, and give the model both sets. The structured-output techniques from the earlier module do this cleanly: ask for a list of sub-questions as JSON, cap it at three, and search them in parallel."}]},{"t":"h2","id":"hyde-and-why-it-sometimes-helps","runs":[{"text":"HyDE, and why it sometimes helps"}]},{"t":"p","runs":[{"text":"Hypothetical document embedding asks the model to write a plausible answer to the question first, then embeds that fabricated answer and searches with it. The mechanism is that passage-to-passage similarity is a better-matched comparison than question-to-passage: a fake answer uses the vocabulary and shape of a real one. Reported gains range from substantial to none, and the reason is visible in the mechanism. When the model knows the domain’s vocabulary, its fake answer resembles the real passage. In a niche domain it fabricates vocabulary that appears nowhere, and the search gets worse. It also adds a generation call of several hundred milliseconds. Try it on your evaluation set; keep it only if it moves the number."}]},{"t":"h2","id":"extracting-filters-from-the-question","runs":[{"text":"Extracting filters from the question"}]},{"t":"p","runs":[{"text":"“Invoices from March for Acme” is a metadata filter wearing the clothes of a query. Ask the model for structured output (a date range, a customer name, a document type) alongside the free-text part, and turn the structured half into a filter on the index. A search restricted to one customer’s March invoices is nearly always better than a similarity search over everything, and this is where the permission-and-metadata work of the next lesson pays back."}]},{"t":"h2","id":"spelling-transliteration-and-language","runs":[{"text":"Spelling, transliteration and language"}]},{"t":"p","runs":[{"text":"Keyword search fails on a misspelling completely; embeddings tolerate small ones. A rewrite step that corrects spelling helps the keyword half of a hybrid system. For a user writing in Hindi or Hinglish against English documents, the rewrite can translate the query, which is often the difference between a hit and nothing. Be honest with yourself about the risk: translation can lose a nuance the user cared about, and a rewrite that changes the meaning is worse than no rewrite. Log both versions so you can see when that happened."}]},{"t":"h2","id":"the-costs-and-a-routing-rule","runs":[{"text":"The costs, and a routing rule"}]},{"t":"p","runs":[{"text":"Every rewrite is a model call: roughly 200 to 500 milliseconds and a fraction of a cent with a small model. Stacked (condense, then expand, then decompose) that is a second and a half before the search has started. So route rather than always rewrite."}]},{"t":"ul","items":[[{"text":"First turn of a conversation, and the message is a complete question: search it as typed."}],[{"text":"Follow-up turn: condense."}],[{"text":"Under five words: expand."}],[{"text":"Contains “and”, “compare”, “versus” or two question marks: consider decomposition."}],[{"text":"Looks like an identifier: do not rewrite at all; a rewrite will helpfully “correct” the invoice number."}]]},{"t":"h2","id":"log-the-rewritten-query","runs":[{"text":"Log the rewritten query"}]},{"t":"p","runs":[{"text":"When retrieval fails you need to know which stage failed. Was the rewritten query wrong, or was the search wrong given a right query? Log the original message, the rewritten query, every sub-query, and the fused candidate list, keyed by request. A failure you cannot decompose into stages is one you will tune blindly, and the rewrite stage is the one that most often turns out to be the culprit when a user reports “it answered a different question”."}]}],"check":{"question":[{"text":"A chat system over product documentation answers first questions well but answers follow-ups like “and for enterprise?” from irrelevant passages. What is the mechanism?"}],"options":[{"text":[{"text":"The embedding model has not been given the query prefix, so follow-up questions embed poorly."}],"correct":false,"tempting":[{"text":"Would degrade every query equally, not specifically the follow-ups."}]},{"text":[{"text":"The chunks are too small to contain both the plan name and the price, so the answer is split across chunks and the model cannot join the two halves back together."}],"correct":false,"tempting":[{"text":"Chunk size would affect first questions as much as follow-ups; the pattern points at the query, not the corpus."}]},{"text":[{"text":"The follow-up is searched as typed, without the topic from the previous turn, so its embedding lands near passages that merely contain “enterprise”."}],"correct":true,"tempting":null},{"text":[{"text":"The model is ignoring retrieved context on later turns because the conversation has grown too long."}],"correct":false,"tempting":[{"text":"Blames generation for a retrieval failure; the wrong passages were retrieved before the model saw anything."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"reranking-and-how-many-chunks","title":"Reranking, and how many chunks to actually send","minutes":9},"next":{"slug":"ingestion-pdfs-tables-and-scans","title":"Ingestion: PDFs, tables and scans, where most retrieval actually fails","minutes":10}}}