The field you assumed was printed

The invoice is machine-printed. Clean template, consistent layout, one of forty suppliers you have seen a thousand times. Accuracy on it is excellent.

Except somebody in receiving wrote the purchase order number in the blank box at the top, in biro. And somebody else initialled the delivery date. Two fields on an otherwise printed document are handwritten, and they are being read by a pipeline tuned and measured on print.

Handwriting is rarely a document type. It is a field, on a document that is otherwise printed, and it is invisible in any accuracy number computed per document.

Why the average hides it

Twelve fields, ten of them printed and read near-perfectly, two handwritten and read at perhaps 70%. Per-document accuracy comes out around 95% and looks fine.

The two weak fields are frequently the operationally important ones, because handwriting appears on documents precisely where a human intervened — the PO number added by receiving, the correction to a quantity, the approval initials. Those are the annotations that carry the process, and they are the ones the pipeline is worst at.

This is a specific case of the general argument for per-field metrics, and it is the case where the distortion is largest.

Detect it as a property of the region

The useful question is not "is this document handwritten" — it is "was this field written by hand", asked per field:

type FieldRead = {
  value: string | null
  /** How the text in this region was produced. */
  script: "printed" | "handwritten" | "mixed" | "unknown"
  confidence: number
  bbox: [number, number, number, number]
}

Two ways to get it, and they are complementary.

Ask the model. A vision model can report whether a region is handwritten, and it is reasonably good at it — this is a visual property rather than a semantic judgement, which is the kind of question models answer reliably.

Look at the OCR confidence distribution. Classical OCR engines report per-word confidence, and handwriting produces a characteristic collapse: uniformly low confidence across a contiguous region, while printed text nearby stays high. A region whose mean word confidence is far below the page mean is very likely not print, and this costs nothing if you are already running OCR for coordinates.

Then route on it

Detection is only worth having if something changes:

function policyFor(field: FieldRead, base: FieldPolicy): FieldPolicy {
  if (field.script !== "handwritten") return base

  return {
    // Hold handwritten fields to a higher bar. The model's confidence
    // is less well calibrated here, so the same number means less.
    threshold: Math.min(0.99, base.threshold + 0.05),
    // And never silently accept a handwritten value on a field that
    // triggers anything irreversible.
    onUncertain: base.onUncertain === "write_flagged"
      ? "write_flagged"
      : "hold_field",
  }
}

The threshold bump is the point. A confidence of 0.92 on printed text and 0.92 on handwriting are not the same claim — the calibration differs, and if you have measured your calibration curve separately for the two you will usually see it directly.

What actually improves accuracy

In order of how much they help.

Crop tightly and send the region alone. A handwritten PO number in a 20×80 pixel box is a tiny fraction of a full page image, and the model is attending to the whole page. Cropping the region and sending it as its own image — at higher effective resolution — is the single biggest improvement available, and it is cheap because the crop is small.

Constrain the output. Most handwritten fields have a known shape. A PO number matching PO-\d{6}, a date, a quantity, a set of known initials. Giving the model the pattern turns an open-ended reading problem into a constrained one:

This field contains a purchase order number matching PO-###### where # is a digit. If the handwriting does not clearly resolve to that pattern, return unreadable rather than the closest guess.

Give it the candidate set where one exists. Initials belong to a known set of approvers. A supplier code is one of forty. Reading handwriting is far easier when the answer must be one of a list, and this converts a transcription task into a classification task — which is a much easier task.

Use both readings. Where you have OCR and a vision model, run both on the cropped region and treat disagreement as a refusal. Agreement across two different systems is a genuinely strong signal; it is the cheapest reliable confidence estimate available for handwriting.

Accept that some of it goes to a person

Handwriting accuracy has a ceiling, and it is below the ceiling for print. Some of it is genuinely illegible — a human cannot read it either, which is worth remembering when a target is being set.

The right response is to plan for it in the operating model rather than to keep tuning:

-- Review load attributable to handwriting. If this is a large share,
-- the fix may be upstream: a form redesign, or a barcode.
select
  count(*) filter (where script = 'handwritten')::float / count(*) as hw_share,
  avg(seconds_to_resolve) filter (where script = 'handwritten')    as hw_seconds,
  avg(seconds_to_resolve) filter (where script = 'printed')        as print_seconds
from review_items
where resolved_at > now() - interval '30 days';

If handwritten fields are 3% of documents and a third of your review load, that is a real number to take to whoever owns the upstream process. Frequently the best available fix is not a better model — it is asking receiving to type the PO number into the portal instead of writing it on the paper, which removes the problem entirely.

That conversation only happens if you can put a number on it, which is the argument for detecting script in the first place.

More in when extraction accuracy collapses, 97% accurate is not a number, and how we build extraction pipelines.