What has to ship with the code

The build is finished, the tests pass, the accuracy target is met, and the repository is handed over.

Four months later the client's team needs to add a field. Nobody knows which prompt file is live, whether the eval suite still runs, what the confidence thresholds were chosen to achieve, or what to do when accuracy drops on a Tuesday. The system works and nobody can touch it.

Conventional software can be understood by reading it. An AI system cannot, because the important decisions are thresholds and trade-offs that are invisible in the code that implements them.

Why this is different from ordinary handover

Read a normal codebase and you can work out what it does. Read an extraction pipeline and you find const HIGH = 0.95 with no indication of what that number was chosen to achieve, what it costs when it moves, or what measurement produced it.

The system's behaviour is defined by a set of choices — thresholds, routing rules, what gets refused, what goes to a human — that were made deliberately against measurements that are not in the repository unless somebody put them there.

The three documents that matter

Beyond the code, the eval set and the harness, three things decide whether the system is maintainable.

1. The decision record

Not architecture. The choices, and the evidence behind each:

## Confidence thresholds

`total` auto-accepts at ≥ 0.98, holds the document below.

**Why 0.98.** Calibration on 200 documents (2026-06-14, prompt a3f81c) showed
the 0.95–0.98 band correct 94% of the time — one escaped error per ~17
documents in that band, and a wrong total posts a wrong payment.
The 0.98+ band was correct 99.6%.

**Cost of moving it.** Down to 0.95: review load falls ~40%, escaped error
rate roughly triples. Up to 0.99: review load rises ~60% for a small gain.

**Recalibrate when.** Any prompt or model change. The bands shift.

That is the difference between a maintainer who can adjust the number and one who is afraid to touch it. It takes ten minutes to write per decision and there are usually fewer than a dozen that matter.

2. The failure runbook

What to do when it breaks, written for whoever is on call rather than for the person who built it:

## Symptom: refusal rate on `issuedOn` jumped from 2% to 14%

**Most likely.** A supplier changed their template. Check which supplier ids
dominate the refusals:
  select supplier_id, count(*) from review_items
  where field = 'issuedOn' and reason = 'unreadable'
    and created_at > now() - interval '3 days'
  group by 1 order by 2 desc limit 5;

**If one supplier dominates.** Their template changed. Add 10 of their recent
documents to the eval set, check whether the prompt handles them, adjust.
Expect ~half a day.

**If it is spread evenly.** Suspect an upstream change — scanner settings, a
new upload path, an image preprocessing regression. Compare source_type
distribution week over week before touching the prompt.

**Do not.** Lower the threshold to make the queue smaller. That converts
visible refusals into invisible errors.

The "do not" section is the most valuable part, because it names the tempting wrong move — and under pressure, the tempting wrong move is what happens.

3. The cost model

What it costs to run, broken down, with the levers:

Current: ~$16 per 1,000 documents all-in (model + review).
  model  $12  — image 1,560 tok, instructions 800 (cached), output 600
  review  $4  — 6% review rate, ~10s per item

Largest levers, in order:
  1. Review interface speed. 6x swing on the review line.
  2. Output field count. Output is 5x input price.
  3. Batch for backfills. Flat 50%.

Watch: Gemini Flash introductory pricing ends 31 Dec 2026 (doubles).
Not currently used here, but relevant if the model is switched.

Without this, the first cost question after handover gets answered by guessing.

The rest of the checklist

• Eval harness in their CI, gating deploys, with the threshold set slightly below current. Not a script somebody runs manually.
• The golden set, with its sampling rationale, and the production failures added during the build. See the eval set is the deliverable.
• Prompt version history, with the eval numbers each version achieved.
• Every credential rotated and held by the client. Not shared, not in a .env in the repository.
• A monitoring dashboard with the four numbers that matter: refusal rate, review queue depth, cost per document, escaped error rate.
• Alerts that fire to their on-call, not ours.
• The known-limitations list. What it does not handle, written down, so the first person to hit one knows it was a decision.

Do the failure drill

The step that separates a handover from a document dump: before the engagement ends, break it on purpose and have the client's team fix it.

Introduce a regression in a branch — a threshold moved, a prompt edited so one field degrades — and have them find it using the harness, the runbook and the dashboard. Two hours.

What it exposes is always the same class of thing: the alert nobody receives because it goes to a channel they do not watch, the runbook query that references a table they cannot read, the eval command that only works with an environment variable that was never written down.

Every one of those would otherwise surface during a real incident, months later, with nobody available who knows the answer.

The test

One question, asked honestly at the end: can somebody who was not on this project add a field, measure whether it worked, and ship it — using only what is in the repository?

If the answer needs a call with us, the handover is not finished. That is not a commercial position, it is a definition — an AI system only one team can operate is a dependency, and the client did not buy a dependency.

More in the eval set is the deliverable, white-label AI work, and seven questions for an AI contractor.