Document-level confidence is nearly useless

An extraction pipeline returns an invoice with confidence: 0.87. Now decide what to do with it.

You cannot. There is no action that follows from 0.87. Route it to a human and you are paying someone to re-read documents that were mostly fine. Accept it and you have accepted whichever single field was wrong. The number describes a document, and a document is not the unit anything downstream cares about.

A document is rarely uniformly good. One number for the whole thing averages a certainty you have with an uncertainty you needed to know about.

What the average hides

A typical scanned invoice: the supplier name is printed at 300 DPI in 14pt and is unambiguous. The invoice number is a clean machine-set string. The date is a rubber stamp at an angle, half over a fold. The total is fine, but the tax line underneath it is a smudge.

Field-level, that document is roughly 1.0, 0.99, 0.42, 0.97, 0.55. Averaged, it is 0.79 — a number that describes none of those fields and hides the two that needed attention. The two bad fields are also the two that end up in an accounts payable ledger.

The shape that works

Confidence rides alongside the value it describes, not beside the document:

type Extracted<T> = {
  value: T
  confidence: number      // 0–1, per field, calibrated
  source: {
    page: number
    bbox?: [number, number, number, number]
  }
}

type Invoice = {
  supplier:      Extracted<string>
  invoiceNumber: Extracted<string>
  issuedOn:      Extracted<PlainDate>
  total:         Extracted<Money>
  tax:           Extracted<Money>
}

source matters as much as confidence. A reviewer told "the tax is uncertain" still has to find the tax. A reviewer told "the tax is uncertain, page 1, this rectangle" is doing a two-second job.

Then route per field, not per document

The point of per-field confidence is that a single document can go to three places at once:

const HIGH = 0.95
const LOW  = 0.70

function route(invoice: Invoice) {
  const fields = Object.entries(invoice) as [keyof Invoice, Extracted<unknown>][]

  const accepted = fields.filter(([, f]) => f.confidence >= HIGH)
  const review   = fields.filter(([, f]) => f.confidence < HIGH && f.confidence >= LOW)
  const rejected = fields.filter(([, f]) => f.confidence < LOW)

  // Write what you trust. Do not hold nine good fields hostage to one bad one.
  commit(invoice.id, accepted)

  // A person confirms these. They see the crop, not the whole document.
  if (review.length) enqueueForReview(invoice.id, review)

  // These are not "low confidence", they are "no answer". Different queue.
  if (rejected.length) enqueueForReextraction(invoice.id, rejected)
}

The middle branch is where the money is. Under a document-level score the whole invoice goes to a person; here one field does, and the reviewer sees a cropped rectangle with a suggested value and two buttons. That is the difference between a 90-second task and a five-second one, at the same accuracy.

Where the number comes from

Three sources, in ascending order of trustworthiness:

Ask the model. Cheap, available everywhere, and badly calibrated — models report high confidence on confidently wrong answers, which is the exact case you built this to catch. Usable as a weak signal, never as a gate on its own.

Token log-probabilities. Where the provider exposes them, the probability mass on the tokens that make up a field is a real measurement of the model's own uncertainty. Better calibrated than self-report because it is not itself a generated claim.

Agreement across samples. Extract the same document two or three times at a non-zero temperature and compare field by field. Fields that agree every time are stable; fields that come back differently are the ones a person should see. It costs you two extra calls and it is the most honest of the three, because disagreement is an observation rather than an opinion.

In practice the third combined with a validation rule beats any of them alone. If line items sum to the total, the total's confidence goes up regardless of what the model said about it — and that check costs nothing, which is the argument made at length in a schema that fails loudly.

Calibrate it or do not publish it

A confidence score nobody has checked is a decoration. The check is a half-day of work:

1. Take 200 documents from your golden set.
2. Bucket every extracted field by reported confidence — 0.9–0.95, 0.95–0.99, and so on.
3. In each bucket, measure how often the field was actually correct.

If the 0.9–0.95 bucket is right 91% of the time, the score means something and you can set thresholds from it. If that bucket is right 60% of the time, your thresholds are fiction and you are shipping a number that gives false comfort to whoever reads the dashboard.

Do this once per prompt version. A prompt change re-rolls the calibration, and a threshold tuned against the old one silently stops working.

More in when extraction accuracy collapses and how we build extraction pipelines.