Rewriting the query, without losing the question

Query rewriting is a standard recommendation and it works. It also, applied uniformly, makes a measurable share of queries worse — and the ones it damages are the ones that were previously working perfectly.

"ERR_4417 on upload" rewritten into "What causes error 4417 during the file upload process and how can it be resolved?" is a fluent, well-formed question that has buried the only token that mattered in fifteen words of padding.

Rewriting helps when the query is underspecified and hurts when it is precise. Applying it to everything trades one failure mode for another.

The two cases it genuinely fixes

Follow-up questions with pronouns. "What about enterprise?" embeds to nothing useful. It only means something given the previous turn, and retrieval has no previous turn. Resolving it against conversation history is not really rewriting — it is completing an incomplete query, and it is close to mandatory in any multi-turn product.

Vocabulary mismatch. A user says "how do I get my money back"; the corpus says "refund eligibility". No shared terms, and embeddings only partly bridge it. Expansion helps here.

Both are cases where the query does not contain enough signal. That is the test.

The case it breaks

Any query built around a rare exact token. Identifiers, error codes, part numbers, names, version strings.

Expansion dilutes them. The rewritten query is longer, so each token contributes less to the vector, and the one token carrying all the information contributes proportionally least. It also breaks lexical search, which was going to find ERR_4417 immediately and now has fourteen other terms competing for weight.

Worse, rewriting is a model call and models paraphrase. ERR_4417 becomes "error 4417", which is a different string and matches nothing in an inverted index.

Gate it

Do not rewrite everything. Decide cheaply, per query:

const IDENTIFIER = /\b([A-Z]{2,}[-_]?\d{2,}|\d+\.\d+\.\d+|[A-Z0-9]{6,})\b/

function shouldRewrite(q: string, history: Turn[]): RewriteMode {
  // Contains something exact. Leave it completely alone.
  if (IDENTIFIER.test(q)) return "none"

  // Depends on prior context. Resolve references — do not expand.
  if (history.length > 0 && hasUnresolvedReference(q)) return "resolve"

  // Short and vague. Expansion is likely to help.
  if (q.split(/\s+/).length < 4) return "expand"

  // A well-formed question of reasonable length. Usually fine as-is.
  return "none"
}

Three modes, not two. resolve and expand are different operations with different risks, and conflating them is why rewriting gets a mixed reputation.

Resolve is safe. Expand is not.

Resolving replaces references with what they refer to. Constrained, low-risk, and the instruction should forbid adding anything:

const RESOLVE_PROMPT = `
Rewrite the user's question so it stands alone without the conversation.
Replace pronouns and references with what they refer to.

Do not add terminology, synonyms, or detail that the user did not supply.
Do not rephrase for fluency. If the question already stands alone, return
it unchanged, character for character.
`

That last sentence matters. Without it, a model asked to rewrite will always rewrite, because returning the input unchanged does not feel like doing the task.

Expanding adds terms the user did not say, which means it can add the wrong ones — the drift failure. A question about refund timing expanded into terminology about refund eligibility retrieves the wrong section confidently.

The safer form of expansion is not to replace the query but to search with both:

// Keep the original. Add the expansion as an additional query.
// Fuse the results, so the original can always win.
const [a, b] = await Promise.all([
  search(original),
  search(expanded),
])
const fused = rrf([a, b])

Reciprocal rank fusion means a chunk found by the original query surfaces even if the expansion drifted. You get expansion's upside without betting the query on it.

Multi-query, and its cost

The stronger version generates several paraphrases and fuses all of them. It measurably improves recall on vague questions.

It also costs a model call plus N searches per query, adds latency on the critical path, and produces diminishing returns past about three variants. Worth it for asynchronous or high-value queries; rarely worth it for an interactive chat where the same latency could have gone to reranking, which generally buys more.

Measure it as a segment, always

Rewriting is the clearest example of a change whose aggregate hides two opposite effects:

                    no rewrite   always   gated
recall@10                 0.78     0.79    0.86
  vague                   0.51     0.74    0.74
  follow-up               0.29     0.81    0.81
  identifier              0.88     0.52    0.88   <- destroyed by "always"
  well-formed             0.91     0.87    0.91

"Always" looks like a one-point improvement and is a disaster on identifier queries. The gated version keeps every gain and none of the damage — and the only way to see any of this is to have tagged the eval set by query type when you built it.

If you take one thing: tag your eval questions by whether they contain an exact identifier. It is the single most informative split in retrieval, and it decides this question and several others.

More in why cosine similarity lies, hybrid search beats a bigger embedding model, and retrieval systems that are actually evaluated.