A retrieval eval you can build in a day

Most teams with a RAG system in production cannot answer a simple question: when the answer is wrong, was it wrong because retrieval missed, or because the model ignored what retrieval gave it?

Those have opposite fixes. Without a retrieval eval you cannot tell them apart, so changes get made on vibes and the system drifts sideways for months.

The eval that answers it is one day of work. Not the sophisticated version — the version that exists.

The eval you have on Friday beats the one you designed on Friday and never built.

What you are building

Fifty questions. For each one, the chunk that ought to answer it. Then a script that runs retrieval and reports how often the right chunk came back in the top k. That is the entire scope.

The metric is recall@k: of the questions where a correct chunk exists, how often is it in the top k results? If recall@10 is 0.55, then no amount of prompt engineering will fix nearly half your failures, because the model is never shown the answer.

Step 1 — get fifty real questions

Real ones. In descending order of value:

1. Production logs. What people actually typed. Best source by a wide margin, because it contains the phrasings you would never think of.
2. Support tickets. Same property, plus they skew towards things that went wrong.
3. Ask the team. Everyone who has demoed the product has three questions they always use. Collect them.
4. Generate them, carefully. Feed a chunk to a model and ask what question this passage answers. Fastest option and the weakest, because the questions inherit the chunk's own vocabulary and are therefore artificially easy. Fine for padding, dangerous as the whole set.

Bias towards questions that are currently failing. An eval set your system aces tells you nothing. Aim for a set the system gets roughly 60–80% right at the start, so there is room to observe both improvement and regression.

Step 2 — label the answer chunk

Sit with the corpus and, for each question, record which chunk contains the answer. This is the boring part and the part that cannot be skipped, and fifty of them is about two hours.

Store it as data, in the repository, in a format a human can edit:

// eval/retrieval.jsonl — one object per line, reviewable in a diff
{"q": "what is the refund window for enterprise customers?", "chunks": ["ent-addendum-4"], "tags": ["policy", "qualifier"]}
{"q": "ERR_4417 on upload", "chunks": ["errors-4417"], "tags": ["identifier"]}
{"q": "how do I get my money back", "chunks": ["consumer-terms-2"], "tags": ["policy", "paraphrase"]}

Two details that pay for themselves:

chunks is a list. Sometimes two chunks would both answer the question. Any of them counts as a hit.

tags exist from the start. They are what lets you segment results later. An aggregate recall number hides the fact that identifier queries are at 0.3 while paraphrase queries are at 0.9, and the segmented view is what tells you what to fix. Adding tags retroactively means re-reading fifty questions.

Step 3 — the harness

import { readFileSync } from "node:fs"
import { retrieve } from "../src/retrieval"

type Case = { q: string; chunks: string[]; tags?: string[] }

const cases: Case[] = readFileSync("eval/retrieval.jsonl", "utf8")
  .trim().split("\n").map((l) => JSON.parse(l))

const K = [1, 5, 10]

async function main() {
  const results = await Promise.all(
    cases.map(async (c) => {
      const hits = await retrieve(c.q, { limit: Math.max(...K) })
      const ids = hits.map((h) => h.id)
      // Rank of the first correct chunk, or Infinity if it never appeared.
      const rank = ids.findIndex((id) => c.chunks.includes(id))
      return { ...c, rank: rank === -1 ? Infinity : rank + 1 }
    }),
  )

  for (const k of K) {
    const hit = results.filter((r) => r.rank <= k).length
    console.log(`recall@${k}  ${(hit / results.length).toFixed(3)}`)
  }

  // MRR: rewards ranking the right chunk first, not merely including it.
  const mrr = results.reduce((n, r) => n + 1 / r.rank, 0) / results.length
  console.log(`MRR       ${mrr.toFixed(3)}`)

  // The segmented view — this is the one that tells you what to do next.
  const tags = [...new Set(results.flatMap((r) => r.tags ?? []))]
  for (const tag of tags) {
    const sub = results.filter((r) => r.tags?.includes(tag))
    const hit = sub.filter((r) => r.rank <= 10).length
    console.log(`  ${tag.padEnd(12)} recall@10 ${(hit / sub.length).toFixed(3)}  (n=${sub.length})`)
  }

  // Every question the system currently cannot answer at all.
  const missed = results.filter((r) => r.rank === Infinity)
  if (missed.length) {
    console.log(`\n${missed.length} total misses:`)
    for (const m of missed) console.log(`  ${m.q}`)
  }
}

main()

Roughly sixty lines, no framework, no service. It prints three numbers, a breakdown, and a list of questions your system cannot answer — and that last list is the most useful output, because it is a work queue.

Step 4 — put it in CI and gate on it

An eval you run when you remember is an eval that stops existing after three weeks.

- name: Retrieval eval
  run: npm run eval:retrieval -- --min-recall-at-10 0.70

Fail the build below the threshold. Set the threshold slightly under where you are today, and raise it as you improve. The purpose is not to hit a number — it is to make a regression impossible to merge without someone consciously lowering the bar in a diff that a reviewer can see.

What it will tell you on day one

Nearly always one of two things, and they point in different directions:

Recall@10 is high — say above 0.85 — and answers are still bad. Retrieval is fine. Your problem is downstream: chunks that are too long for the qualifier to survive, a prompt that lets the model answer from prior knowledge, or no instruction to abstain when the context does not cover the question.

Recall@10 is low — below 0.6. Stop touching the prompt entirely. Nothing you do there matters while the model is not being shown the answer. Go and look at the segmented output; in most corpora the identifier tag is dragging the average down and hybrid search is the next thing to try.

Either way you now know which half of the system to work on, which is the entire point and is worth considerably more than a day.

More in how to evaluate an LLM pipeline, why your RAG returns wrong answers, and retrieval systems that are actually evaluated.