Rewriting the question before you search it

The query the user typed is not the query to search

A user asks “what does the Pro plan cost?”, gets an answer, and then types “and for enterprise?”. Embed that second message on its own and it lands nowhere useful. The word that matters, cost, is in the previous turn. The retrieval system, which sees only the string it was given, returns chunks about enterprise onboarding, enterprise security, anything with the word in it, and the model answers from the wrong passages.

This is the most common failure in chat-over-documents, and it has nothing to do with embeddings. The fix is to rewrite the message into a standalone query before searching. A small, fast model, given the last few turns and asked to produce one self-contained search query, returns “enterprise plan pricing” in a hundred milliseconds for a fraction of a cent. That one step fixes more follow-up questions than any change to the index.

REWRITE = """Given the conversation and the latest message, write one
standalone search query that captures what the user is asking.
Output the query only."""

Expansion

Terse queries miss. “refund” against a corpus that says “returns and reimbursement” may never meet. Ask the model for three paraphrases, search all four, and fuse the lists with reciprocal rank fusion from the hybrid-search lesson. Recall goes up on short queries; the price is three extra searches, which are cheap, and one extra model call, which is not free in latency. Reserve it for queries under a handful of words.

Decomposition

“Compare the refund policy for annual and monthly plans” needs two facts that live in two places. One search with that sentence finds whichever place matches the sentence better and misses the other. Decompose into two sub-queries, search each, and give the model both sets. The structured-output techniques from the earlier module do this cleanly: ask for a list of sub-questions as JSON, cap it at three, and search them in parallel.

HyDE, and why it sometimes helps

Hypothetical document embedding asks the model to write a plausible answer to the question first, then embeds that fabricated answer and searches with it. The mechanism is that passage-to-passage similarity is a better-matched comparison than question-to-passage: a fake answer uses the vocabulary and shape of a real one. Reported gains range from substantial to none, and the reason is visible in the mechanism. When the model knows the domain’s vocabulary, its fake answer resembles the real passage. In a niche domain it fabricates vocabulary that appears nowhere, and the search gets worse. It also adds a generation call of several hundred milliseconds. Try it on your evaluation set; keep it only if it moves the number.

Extracting filters from the question

“Invoices from March for Acme” is a metadata filter wearing the clothes of a query. Ask the model for structured output (a date range, a customer name, a document type) alongside the free-text part, and turn the structured half into a filter on the index. A search restricted to one customer’s March invoices is nearly always better than a similarity search over everything, and this is where the permission-and-metadata work of the next lesson pays back.

Spelling, transliteration and language

Keyword search fails on a misspelling completely; embeddings tolerate small ones. A rewrite step that corrects spelling helps the keyword half of a hybrid system. For a user writing in Hindi or Hinglish against English documents, the rewrite can translate the query, which is often the difference between a hit and nothing. Be honest with yourself about the risk: translation can lose a nuance the user cared about, and a rewrite that changes the meaning is worse than no rewrite. Log both versions so you can see when that happened.

The costs, and a routing rule

Every rewrite is a model call: roughly 200 to 500 milliseconds and a fraction of a cent with a small model. Stacked (condense, then expand, then decompose) that is a second and a half before the search has started. So route rather than always rewrite.

• First turn of a conversation, and the message is a complete question: search it as typed.
• Follow-up turn: condense.
• Under five words: expand.
• Contains “and”, “compare”, “versus” or two question marks: consider decomposition.
• Looks like an identifier: do not rewrite at all; a rewrite will helpfully “correct” the invoice number.

Log the rewritten query

When retrieval fails you need to know which stage failed. Was the rewritten query wrong, or was the search wrong given a right query? Log the original message, the rewritten query, every sub-query, and the fused candidate list, keyed by request. A failure you cannot decompose into stages is one you will tune blindly, and the rewrite stage is the one that most often turns out to be the culprit when a user reports “it answered a different question”.