How to evaluate an LLM pipeline

Most teams know they should have evals and do not, because every guide starts by describing a research-grade harness that would take a quarter to build.

This is the smallest version that changes how you work. It takes about a week and you will use it every day afterwards.

An eval set is not a research artefact. It is a regression test that happens to be about language.

Step 1 — the golden set

Fifty examples. Not five hundred. Fifty is enough to catch real regressions and small enough that you will actually build it.

Each row needs three things:

• The input — a real one, taken from production or from support tickets. Not one you invented, because the ones you invent are the ones you already handle.
• The correct output — what a competent human would produce.
• Where the answer comes from — the document, the field, the passage. For retrieval systems this is what lets you separate recall failures from generation failures, which is the whole game.

Bias the set toward the hard cases. A set of fifty easy examples will sit at 98% forever and tell you nothing. Include the weird ones, the ambiguous ones, and three or four that should return "I don't know".

Step 2 — measure the stages separately

The single most common mistake is one number for the whole pipeline. When it drops you learn that something broke, and nothing about what.

That last row matters more than people expect. A system that never abstains has 100% coverage and lies for the last 8%.

Step 3 — grading

Three options, in increasing order of cost:

Exact match. For extraction and classification. Cheap, deterministic, unambiguous. Use it wherever the output shape allows.

Fuzzy / semantic match. For short free text. Embedding similarity against the reference above a threshold. Adequate, and its failure mode is being too generous.

LLM-as-judge. For long-form answers. Works, with two rules that are not optional: give the judge the reference answer rather than asking it to assess quality in the abstract, and spot-check the judge against human grading on twenty examples before you trust it. An unvalidated judge is a random number generator with good manners.

Step 4 — put it in CI

This is the step that converts an eval set from a document into a practice.

• It runs on every pull request that touches a prompt, a model version, a chunking parameter, or the retrieval config.
• It fails the build on a regression past a threshold you agreed in advance.
• It prints the diff — which examples changed, not just the aggregate.

The diff is the part people skip and the part that pays. Aggregate score moved from 0.86 to 0.84 is not actionable. "These four examples now fail, and all four are multi-page documents" is a bug report.

Step 5 — treat production failures as new rows

Every time a customer reports a wrong answer, it becomes a row in the golden set before it gets fixed. This is the flywheel: the set becomes a precise map of the ways your specific system fails, and regressions of previously-fixed bugs become impossible to ship.

What this costs

About a week for a first version, and the first afternoon usually surfaces something surprising. It is the cheapest engineering on this list and the one most consistently skipped, because it produces no visible feature.

The honest pitch for doing it: without evals you cannot safely change anything, so the system freezes. Every team that has been afraid to touch a prompt for three months is describing a measurement problem.

Related: why your RAG returns wrong answers and our approach to retrieval.