Teach the extractor to refuse

Ask a model for the total on a document that has no total, and you will get a total. Ask for the total on a page so degraded that no human could read it, and you will get a total then too. It will be a number of the right magnitude, in the right currency, formatted correctly.

This is not the model malfunctioning. You gave it a schema with a required numeric field and no way to decline, and it filled the field. The pipeline asked a question that had no valid non-answer.

If the only shapes your output can take are "a number" and "a different number", every failure is silently converted into a plausible value.

Three states, not two

Most pipelines model extraction as succeeded or crashed. Real documents need four outcomes, and three of them are not failures:

• Found. The field is in the document and was read.
• Absent. The field is genuinely not in this document. A receipt with no VAT line is not an error; a lot of receipts have no VAT line.
• Unreadable. The field is there and could not be read — obscured, cut off, too degraded.
• Ambiguous. Two readings are defensible. The date case, mostly.

Collapsing absent and unreadable into one bucket is the expensive mistake. They lead to completely different actions: absent means write null and move on, unreadable means fetch a better scan or put a person on it. A pipeline that cannot distinguish them either chases people for documents that were fine, or writes null over data that was really there.

Make refusal a value the type system understands

type Field<T> =
  | { status: "found";      value: T; confidence: number; page: number }
  | { status: "absent" }
  | { status: "unreadable"; reason: string; page: number }
  | { status: "ambiguous";  candidates: T[]; page: number }

type Receipt = {
  merchant: Field<string>
  total:    Field<Money>
  vat:      Field<Money>     // legitimately absent much of the time
  paidOn:   Field<PlainDate>
}

A discriminated union does the enforcement for you. Downstream code cannot read receipt.total.value without first proving the status is found, so the "undefined sailed through into the ledger" bug becomes a compile error rather than a support ticket.

Getting the model to actually use it

Two things have to be true, and most implementations only do the first.

The schema must permit refusal. If total is a required number, the model has no legal way to decline. Model it as the union above and the refusal is representable.

The prompt must make refusal safe. Models are trained to be helpful, and returning nothing reads as unhelpful. Say so explicitly, and say what the alternative costs:

For each field, return absent if the field does not appear in this document, and unreadable if it appears but you cannot read it with confidence. A refusal is a correct answer and is preferred to a guess. Guessed values are more expensive to us than missing ones, because a missing value is visible and a wrong value is not.

That last sentence changes behaviour more than any other line in the prompt. It supplies the cost asymmetry the model would otherwise have to infer, and it is also just true.

Watch the refusal rate as a metric

Once refusal is possible, the refusal rate becomes one of the most informative numbers you have — and it moves for reasons worth knowing about:

select
  date_trunc('day', extracted_at) as day,
  field,
  count(*) filter (where status = 'unreadable')::float / count(*) as refusal_rate
from extractions
group by 1, 2
order by 1 desc;

A refusal rate near zero means the model is not really using the option, and you are back where you started. A rate that climbs on one field means something upstream changed — a supplier redesigned a template, a scanner setting drifted, someone started uploading phone photographs. That is a genuine early warning signal, and it arrives days before anyone downstream notices a data quality problem.

Set the alert on the derivative, not the level. Five per cent unreadable on a handwriting field is normal. Five per cent on a field that was at nought point two last week is an incident.

What it costs

Refusals go somewhere, and that somewhere is usually a person. Before turning this on, work out the two numbers:

• Cost per review. A reviewer at $25/hour resolving a cropped single field in ten seconds is about seven cents.
• Cost per undetected error. This is the one people skip, and it is the reason the feature exists. A wrong total in a payables ledger costs a correction, a reconciliation, and some amount of trust.

If the second number is smaller than the first, you do not need this and you should not build it. That is a real answer for some domains. For anything financial, medical, or legal, the second number is larger by two or three orders of magnitude, and the review queue is obviously worth it.

More in when extraction accuracy collapses and how we build extraction pipelines.