Your golden set is your happy path

Someone is asked to assemble test documents. They open the folder, pick fifty that clearly show what the system is supposed to do, and label them carefully.

Every choice in that sentence is reasonable and the result is a set that overstates your accuracy by ten or twenty points. Documents get picked because they are legible, and legible documents are the ones that were never going to fail.

You cannot sample a test set by choosing. Choosing is the bias.

Sample by strata, not by judgement

The fix is to define the dimensions along which documents vary, then sample deliberately from each — including the cells you would never have picked.

For document extraction, the dimensions that matter are usually:

-- Stratified sample. Take n from each cell, including the ugly cells.
with strata as (
  select
    id,
    source_type,                                    -- scan | photo | native pdf
    case when page_count = 1 then '1' 
         when page_count <= 3 then '2-3' 
         else '4+' end                as pages,
    coalesce(supplier_id, 'unknown')  as supplier,
    case when quality_score < 0.4 then 'poor'
         when quality_score < 0.7 then 'fair'
         else 'good' end              as quality,
    row_number() over (
      partition by source_type, supplier_id
      order by random()
    ) as rn
  from documents
  where received_at > now() - interval '90 days'
)
select * from strata where rn <= 3;

Two properties this has that hand-picking does not.

It includes documents nobody would choose. The photograph taken at an angle, the fourth-generation scan, the supplier with the strange template. These are where accuracy actually lives.

It is reproducible. A stratified query can be re-run next quarter to check whether the strata themselves have shifted, which is a different and equally useful question.

Weight it towards the tail on purpose

A representative sample is not always what you want. If 85% of production is clean native PDFs, a representative 100-document set has 85 easy documents telling you nothing and 15 hard ones carrying all the signal.

Over-sample the hard strata, and record the true production weights so you can compute both numbers:

// Measured on the eval set, reported two ways.
const evalAccuracy = weighted(results, uniformWeights)      // per-stratum view
const prodAccuracy = weighted(results, productionWeights)   // what users see

The per-stratum number tells you where to work. The production-weighted number is what you quote to a customer. Reporting only the first flatters you; reporting only the second hides which stratum is broken.

Label the answer, not the output

The expensive mistake is generating a label from the current pipeline and correcting it. That anchors every label to what the system already does, and it systematically fails to catch fields the pipeline never attempts.

Label from the document. For each field record the correct value, and where the value is not present or not readable record that, using the same vocabulary the pipeline uses — the refusal states:

{
  "documentId": "inv-8823",
  "stratum": { "source": "photo", "pages": "2-3", "quality": "poor" },
  "fields": {
    "total":     { "status": "found", "value": { "amount": 124050, "currency": "EUR" } },
    "issuedOn":  { "status": "ambiguous", "candidates": ["2026-04-03", "2026-03-04"] },
    "vat":       { "status": "absent" },
    "poNumber":  { "status": "unreadable", "note": "obscured by staple" }
  }
}

That ambiguous entry is doing real work. Without it, a pipeline that correctly flags an ambiguous date scores as a miss against a label that arbitrarily picked one reading — and you optimise towards confident guessing, which is the opposite of what you want.

Keep it honest over time

Golden sets decay. Suppliers change templates, scanner fleets get replaced, customers in a new region start uploading documents in a different format. A set assembled in March describes March.

Three habits keep it alive:

Refresh a slice quarterly. Replace 20% with a fresh stratified sample. Total churn destroys comparability; zero churn means measuring an archive.

Add every production failure. When the review queue surfaces something the pipeline got wrong, that document belongs in the set. This is the cheapest source of genuinely hard cases and it costs nothing to collect — the labelling was already done by the reviewer.

Watch for stratum drift. If the production mix moves — photographs from 5% to 30% because a mobile upload feature shipped — your production-weighted accuracy changed without the pipeline changing:

-- Run monthly. A moving distribution invalidates your weights.
select source_type, count(*)::float / sum(count(*)) over () as share
from documents
where received_at > now() - interval '30 days'
group by 1;

Size it usefully

A hundred documents with per-field labels is a good target for a first set. That is enough for meaningful per-stratum numbers and small enough to label in a day or two.

Be honest about what that buys statistically. On 100 documents, a measured accuracy of 94% has a 95% confidence interval of roughly ±4.5 points. So a change from 94% to 96% is not a result — it is noise. Treat differences under about five points as unresolved and either gather more documents or run the comparison on the specific stratum you think moved.

That constraint is a feature. It stops the team from shipping prompt changes on the strength of two extra documents passing, which is otherwise a very easy trap to fall into when the eval is the only feedback available.

More in how to evaluate an LLM pipeline, when extraction accuracy collapses, and how we build extraction pipelines.