GuidesDiagnostic9 min read

When extraction accuracy collapses in production

It worked on your test set and fails on real documents. The distribution gap, how to measure it, and what to do that isn't 'try a bigger model'.

Written for

“It works in testing and falls apart on real customer documents.”

It hit 96% on your test set. It is visibly worse in production and nobody can say by how much. This is the most common failure in applied extraction and it is almost never fixed by a bigger model.

Your test set was built from documents you chose. Production sends documents you did not.

The distribution gap

When a team assembles test documents, they pick ones that demonstrate the problem: clear scans, representative layouts, complete fields. They are not being careless — those are the documents that make the requirement legible.

Production sends the long tail:

Photographs, at an angle, with a shadow and part of a desk in frame
Second and third pages that assume context from page one
A currency you did not plan for, and a decimal separator that means the opposite of what you assumed
Handwriting in a field you assumed was printed
A supplier who redesigned their template last month
Scans of printouts of PDFs, three generations from anything digital
Documents that are the wrong document entirely, submitted by mistake

Each of those is a different failure. Averaged into one accuracy number, they are invisible.

Measure the gap before fixing anything

Pull two hundred random production documents. Random — not sampled by anyone who knows what the system handles well. Grade them by hand. It takes a day.

Then compute accuracy per segment, not overall:

by document source or supplier
by whether it was digital or scanned
by page count
by whether every expected field was present

You will almost always find that overall accuracy is a blend of one segment at 99% and one at 60%, and that the 60% segment is a specific, nameable, fixable population.

That is a much better problem to have than "accuracy is bad".

The four fixes, in order of return

1. Make failure visible

Before improving accuracy, make wrong answers detectable. Per-field confidence, persisted alongside the value, with a threshold that routes low-confidence records to review rather than into the table of record.

This does not improve accuracy at all. It converts silent errors into a queue, which is the difference between a problem you are managing and one that is compounding.

2. Validate what can be validated

A large share of extraction errors violate a rule you already know:

Line items must sum to the subtotal
Dates must be plausible and ordered
Identifiers have checksums — VAT numbers, IBANs, ISBNs
Totals are positive; quantities are integers

Cross-field arithmetic is the highest-return validation in document extraction and most pipelines have none. A hallucinated figure rarely satisfies the sum.

3. Fix the input, not the model

Deskew, denoise, upscale low-DPI scans, split multi-document PDFs, detect and reject the wrong-document case explicitly. Preprocessing routinely recovers more accuracy than any model change, and it is cheap and deterministic.

4. Then, and only then, the model

Better prompting with two worked examples of your hard cases. A schema that fails loudly. Escalating low-confidence pages to a larger model rather than sending everything to it.

Notice this is fourth. Teams start here, because it is the lever that feels like progress.

What good looks like

Not 100%. A pipeline that is honest is better than one that is confident:

A stated accuracy figure per segment, measured on real documents
A confidence threshold with a review queue below it
Validation rules that catch the arithmetic-violating errors for free
A stored trace, so any specific record can be explained
A regression suite that fails the build when a change makes it worse