Perfect retrieval, wrong answer

You built the retrieval eval. Recall@10 is 0.95. The right chunk is in the context on nineteen queries out of twenty.

Users are still complaining.

This is the good problem, and it is a different problem. Retrieval is no longer the constraint, which means every hour spent on chunking, embeddings and reranking from here is an hour spent on the part that already works.

Recall measures whether the answer was available. It says nothing about whether the model used it, used only it, or noticed that it did not fully cover the question.

Four failures downstream of retrieval

The model answered from prior knowledge. The correct chunk was there and the model produced a generic, plausible answer from training data instead — often subtly different from your actual policy. Most common when the retrieved passage is hedged or awkwardly phrased and the model's own version reads more fluently.

The answer was diluted. Ten chunks in context, one relevant. The model blended them, or anchored on the most confidently-phrased passage rather than the correct one. Recall@10 counts this as a success; the user got an average of ten documents.

Partial coverage went unacknowledged. The context answered half the question. The model answered the whole thing, inventing the other half, with a citation attached to the half that was real.

A conflict was resolved silently. Two retrieved chunks disagree — an old policy and a new one, or two regional variants. The model picked one and said nothing. Users notice this one, because they know both exist.

Tell them apart with an ablation

The diagnostic is cheap and it is decisive. Take the failing queries and run each one three ways:

for (const q of failures) {
  const results = {
    // 1. Normal. The baseline failure.
    normal: await answer(q.text, await retrieve(q.text)),

    // 2. Oracle context: ONLY the labelled correct chunk, nothing else.
    oracle: await answer(q.text, [q.goldenChunk]),

    // 3. No context at all.
    blind:  await answer(q.text, []),
  }
  await record(q.id, results)
}

Read the grid:

That first row is the most common result, and its fix is counterintuitive: retrieve fewer chunks. Teams increase topK when answers are bad, which raises recall and lowers answer quality simultaneously.

Stop the model answering from memory

The single highest-value prompt change, and it has to be explicit:

Answer using only the provided context. If the context does not contain the answer, say so and stop. Do not supplement the context with your own knowledge, even where you are confident it is correct. An incomplete answer that stays within the context is better than a complete one that leaves it.

Then verify it rather than trusting it — citation verification catches claims that cannot be traced to a retrieved span, and the rate at which that check fires is a direct measurement of whether the instruction is holding.

Make conflict a visible outcome

Retrieved chunks disagree more often than people expect, and silently picking one is the worst available handling:

type GroundedAnswer =
  | { kind: "answered"; text: string; claims: Claim[] }
  | { kind: "not_covered"; text: string }
  | { kind: "conflicting"; text: string; positions: Position[] }

The conflicting branch is genuinely useful output. "The consumer terms say 30 days; the enterprise addendum says 90. Which applies depends on your agreement type." That is a better answer than either alone, and it is only possible if the model is given a way to say it.

Note that most conflicts are better prevented than reported — filtering on status and audience removes the superseded and inapplicable documents before they can conflict. Reporting is the fallback for genuine ambiguity.

Measure end-to-end, separately

Retrieval and answer quality need different harnesses, because they fail independently:

retrieval:   recall@k, MRR                  — is the answer available?
grounding:   citation verification rate     — is the answer traceable?
             fabricated-citation rate       — did it invent support?
coverage:    abstention rate on unanswerable questions
answer:      graded correctness             — is it right?

The abstention row needs unanswerable questions in the eval set, and almost no eval set has them. Add ten questions your corpus genuinely does not answer. A system that answers all ten is fabricating, and no other metric on that list will show it.

The order to work in

1. Recall below 0.8? Fix retrieval. Nothing downstream matters yet.
2. Recall above 0.9 and answers bad? Run the ablation above. It takes an hour and it tells you which of four problems you have.
3. Dilution? Retrieve fewer, or rerank and pass the top three or five instead of ten.
4. Prior knowledge? Tighten the prompt and enforce citations.
5. Coverage or conflict? Give the model the vocabulary to report them.

The mistake worth avoiding is doing this in the other order — spending three weeks on embeddings and chunk sizes because retrieval is the part with the tuneable knobs, when the ablation would have said on day one that retrieval was fine.

More in a retrieval eval you can build in a day, why your RAG returns wrong answers, and how to evaluate an LLM pipeline.