The review queue is the product

Any extraction system with a real accuracy requirement has a human in it somewhere. Not as a failure of the automation — as the design. Ninety-five per cent automated with a fast path for the rest beats ninety-nine per cent automated with no path at all, because the missing one per cent is where the expensive errors live.

The queue where those land is usually built last, in a hurry, and it decides whether the whole economics work.

Nobody buys "97% accurate". They buy "the 3% costs four seconds each", which is a statement about your review interface, not your model.

The arithmetic that decides everything

Before any design work, two numbers:

cost per review     = reviewer hourly rate ÷ reviews per hour
cost per escaped error = correction + reconciliation + relationship damage

At $25/hour, the review cost is entirely a function of how long one item takes:

That is a 15× spread, and it is decided by interface design rather than by anything about the model. At 100,000 documents a month with a 3% review rate, 3,000 reviews at 60 seconds costs $1,250 a month; at four seconds it costs $84.

The four-second review is achievable, and the rest of this is how.

Review the field, not the document

The single biggest lever. A document-level queue shows a reviewer an entire invoice and asks "is this right?" — which requires reading every field to answer, most of which were fine.

A field-level queue shows one uncertain field. This depends on per-field confidence; with a single document-level score there is nothing to narrow to.

create table review_items (
  id            bigserial primary key,
  document_id   text not null,
  field         text not null,          -- 'total', 'issuedOn'
  extracted     jsonb,                  -- what the model said, nullable on refusal
  confidence    numeric(4,3),
  reason        text not null,          -- low_confidence | ambiguous
                                        -- | validation_failed | unreadable
  -- What the reviewer needs on screen. Precomputed, never derived at render.
  page          int not null,
  bbox          int[4],                 -- crop rectangle in page coordinates
  candidates    jsonb,                  -- alternatives, for one-click selection

  status        text not null default 'open',
  resolved_by   text,
  resolved_at   timestamptz,
  final_value   jsonb,
  -- Did the reviewer accept what the model proposed? This is your accuracy
  -- measurement, produced for free by the act of reviewing.
  was_correct   boolean
);

create index on review_items (status, confidence)
  where status = 'open';

What the reviewer sees

Three elements, and nothing else:

1. The crop. The bbox rectangle from the page, at legible resolution, with a little surrounding context. Not the whole page — a crop the eye lands on without searching.
2. The proposed value, large, and the candidates where they exist.
3. Two keys. Enter to accept, or type a correction. No mouse.

Sorting matters more than it sounds. Group open items by field, not by document, so a reviewer does 200 consecutive date confirmations rather than alternating between dates, totals and supplier names. Same task repeated is several times faster than the same tasks interleaved, and it is a order by field, confidence away.

The corollary: never make a reviewer open the source PDF. The moment the interface requires finding the value in a document viewer, you are at sixty seconds, and every design decision that leads there costs 15×.

The queue measures your accuracy for free

was_correct turns the review queue into a continuously refreshed accuracy measurement over exactly the distribution you are actually seeing — which is better than a golden set assembled six months ago:

-- Precision within each confidence band. This is your calibration curve,
-- recomputed nightly from work someone was doing anyway.
select
  width_bucket(confidence, 0, 1, 20) / 20.0 as band,
  count(*)                                  as reviewed,
  avg(was_correct::int)                     as actually_correct
from review_items
where status = 'resolved' and resolved_at > now() - interval '30 days'
group by 1 order by 1;

If the 0.90–0.95 band comes back correct 91% of the time, your thresholds are sound. If it comes back at 60%, your auto-accept threshold is admitting errors and should move up today. This is also the data that tells you when a prompt change helped, because it is measured against real traffic rather than a curated set.

Two failure modes to design against

The queue outgrows the reviewers. If arrivals exceed throughput, the backlog grows without bound and eventually someone clears it by bulk-accepting, which converts your safety mechanism into a rubber stamp. Monitor queue depth, alert on the trend, and when it crosses a ceiling, raise the auto-accept threshold deliberately and record that you did — a conscious accuracy trade-off is fine, a silent one is not. This is the same shape as bounding an approval queue.

Reviewer drift. People who confirm hundreds of items an hour start accepting by reflex. Salt the queue with a small percentage of items whose answer you already know, and track per-reviewer accuracy on those. Not to discipline anyone — to detect when the interface has become too easy to click through, which is a design problem that presents as a people problem.

When to skip all of this

If the cost of an escaped error is genuinely lower than three cents, do not build a review queue. Write the low-confidence value, flag it, and let the correction happen downstream where somebody notices.

That is a real answer for a lot of internal tooling. It is not the answer for anything financial, medical, or legal, where the second number in the opening arithmetic is three orders of magnitude larger than the first and the queue pays for itself in the first month.

More in document-level confidence is nearly useless, teach the extractor to refuse, and how we build extraction pipelines.