Rewriting the query before you search

The string you search with is a design decision

The user typed “what about the second one?”. You cannot search with that. It contains no content words, and the thing it refers to is three turns above it in a conversation the retriever has never seen.

This is the most common retrieval failure in conversational products and it is not a retrieval failure at all. The retriever did its job on the string it was given. The string was wrong.

Four transformations sit between a user’s message and a good query, and each solves a different problem.

1. Resolve the conversation

Rewrite the latest message into a standalone question using the last few turns.

Given this conversation, rewrite the final user message as a single
self-contained question. Keep the user's own terminology. If it is already
self-contained, return it unchanged.

“What about the second one?” becomes “What is the refund window for the standard plan?”. Now it can be embedded, keyword-searched and cached.

Two details that matter. Keep the user’s own vocabulary: a rewrite that translates their words into your internal jargon can search a corpus that uses the user’s words and lose. And return the original unchanged when it is already standalone, because rewriting a good query is a chance to make it worse for no benefit.

2. Decompose the compound question

“Compare the refund policies for the standard and enterprise plans.” One embedding of that sits somewhere between the two topics and may retrieve neither well.

Split into sub-queries, retrieve for each, merge. It costs one extra model call and roughly doubles the retrieval work, and it is the difference between a comparison built on both sources and one built on whichever happened to rank higher.

The signal for when to do it: questions containing “and”, “versus”, “compare”, “both”, or two named entities. A cheap classifier or a short prompt catches most of them, and you can skip decomposition on the rest.

3. Expand and fuse

Generate three or four paraphrases of the query, retrieve for all of them, and fuse the ranked lists. This raises recall because different phrasings match different vocabulary in the corpus, and it does it without needing the corpus to be re-indexed.

The fusion is the same reciprocal-rank method the hybrid search lesson uses next. The cost is n times the retrieval work, which is usually cheap compared with the generation call.

4. HyDE: search with a hypothetical answer

Ask the model to write the answer it thinks is correct, then embed that and search with it.

The reasoning is that a question and its answer are not lexically or distributionally alike: a question is short, interrogative and vague; an answer is declarative and uses the domain’s vocabulary. Searching with a fabricated answer moves the query vector into the region where real answers live.

It works, sometimes strikingly, on domains where the model has genuine background knowledge. It fails on private material the model knows nothing about: the hypothetical answer is then a confident invention that pulls retrieval towards documents about a thing that does not exist in your corpus. Always fuse HyDE’s results with the original query’s rather than replacing them, so a bad hypothesis costs you nothing.

The cost you are adding

Every one of these is a model call before retrieval, which means latency in front of the user’s first token. A small fast model does rewriting well; this is not a task that needs your best one.

Budget honestly: a rewrite call at 300 milliseconds plus retrieval at 80 milliseconds plus generation is a noticeably slower product than retrieval plus generation. Three mitigations, in order of usefulness:

• Skip it when it is not needed. First message of a conversation, no pronouns, more than four content words: search directly. Most queries need no rewriting.
• Cache rewrites keyed on the conversation tail. Repeated and near-repeated questions are common.
• Run rewriting in parallel with a direct search of the raw query, and fuse. The direct result is free and often good.

Keep the original in the mix

The rule that prevents the whole class of rewriting disasters: never discard the user’s original query. Rewriting adds a hypothesis; it should not replace the evidence.

A rewrite can invent a constraint that was not there (“for orders placed in India”), narrow a broad question, or resolve a pronoun to the wrong antecedent. All three produce a confidently wrong retrieval set. Fusing the original query’s results in costs one more search and bounds the damage.

What to log

Log the original message, the rewritten query, and which of them produced each retrieved passage. When somebody reports a bad answer, this one line separates three otherwise indistinguishable failures: a bad rewrite, a good rewrite that retrieved badly, and a corpus that does not contain the answer.

Without it, the rewritten query is invisible, and you will spend the investigation looking at a question the retriever never saw.