Why your RAG system returns wrong answers

When a retrieval system answers wrongly, the reflex is to blame the model and start rewriting the prompt. That is almost always the wrong layer.

A RAG system is four systems in a trenchcoat. Three of them fail silently, and only the fourth gets blamed.

The four are: chunking, retrieval, ranking, and generation. A wrong answer comes from one of them specifically, and the fix for each is different. Here is how to tell them apart.

The first diagnostic, and it takes ten minutes

Before changing anything, answer one question:

Was the correct passage in the context the model received?

Take ten questions it got wrong. For each, log what was retrieved and read it yourself.

• The right passage was not there — you have a retrieval or chunking problem. The model was asked to answer from material that did not contain the answer, and it did what models do. No prompt fixes this.
• The right passage was there and the answer is still wrong — now you have a generation problem, and prompting is a reasonable lever.

In our experience, the split is roughly 80/20 in favour of the first case. Most "hallucination" bugs are recall failures wearing a costume.

The six causes

1. The chunk boundary cut the answer in half

Fixed-size chunking splits mid-table, mid-sentence, mid-clause. The retrieved chunk contains the question's subject and none of its predicate.

Symptom: the right document is retrieved and the answer is confidently incomplete. Fix: chunk on document structure — headings, sections, table boundaries — and overlap. Then re-run your evals, because chunk size interacts with everything.

2. Embedding similarity is not relevance

Vector search finds topically similar text. "What is the warranty period on the X-40?" is topically similar to every warranty paragraph you have, including all the ones for other products.

Symptom: fails on specific identifiers, part numbers, dates, names — and looks fine on general questions. Fix: hybrid search. Run lexical (BM25) alongside vector and fuse the results. This is the single highest-return change for most systems, and it is a day of work.

3. No reranking

Your top-20 candidates contain the right passage at position 9. You put the top 5 in the context. The answer was retrieved and then discarded.

Symptom: recall@20 is good, recall@5 is bad. If you have never measured both, measure them today. Fix: a cross-encoder reranker over the candidate set. Slower per query, and usually worth it.

4. The index is stale

Source documents changed. Nothing re-embedded them. The system is confidently answering from a version that no longer exists.

Symptom: it was right last quarter and is wrong now, on questions it used to handle. Fix: re-embed on source change, and store the source's last-modified timestamp alongside each vector so you can prove freshness rather than assume it.

5. There is no abstention path

The system is architecturally incapable of saying "I don't know". Given five irrelevant chunks, it will still produce a fluent answer, because that is what it was asked to do.

Symptom: it never says it is unsure, including when it should be. Fix: a relevance threshold on the retrieval scores, and an explicit instruction plus a path for returning "not found in the documents". Then make sure the UI actually shows that state instead of hiding it.

6. Nobody is measuring anything

The deepest cause. Without a golden set, every change is evaluated by someone trying three questions they happen to remember, which means regressions ship undetected and improvements cannot be told from luck.

Fix: see how to evaluate an LLM pipeline. Build the harness before you fix anything else, or you will not know whether the fixes worked.

A one-week order of operations

1. Day 1 — build a golden set of 50 real questions with known answers and the passage each should come from.
2. Day 2 — measure recall@5 and recall@20 separately from answer quality. This will probably tell you the answer immediately.
3. Day 3 — add hybrid lexical + vector search. Re-measure.
4. Day 4 — add reranking if recall@20 was much better than recall@5. Re-measure.
5. Day 5 — only now, touch the prompt.

The order matters. Every team that starts at day 5 spends a month there.

If you would rather have someone do this pass with you on a system that is already in front of customers, that is exactly what the audit covers, and it is fixed price.