You changed the prompt. Now what about the last 400,000 documents?

The prompt now handles a supplier template that used to fail. Accuracy on the golden set went from 94% to 97%. Ship it.

Behind that improvement sits every document already processed under the old prompt. Some of them contain the errors the change just fixed. They are in the database, they look identical to correct records, and nothing marks them.

A prompt change is a schema migration for data you have already produced. It just does not come with a migration file.

Version everything that touched the record

You cannot decide what to reprocess if you cannot tell what produced what:

alter table extractions add column prompt_version text not null;
alter table extractions add column model         text not null;
alter table extractions add column schema_version text not null;
alter table extractions add column extracted_at  timestamptz not null;

prompt_version should be a content hash of the actual prompt template, not a hand-incremented number. Hand-maintained versions get forgotten during exactly the small edits that change behaviour:

// Derived, so it cannot drift from what was actually sent.
export const PROMPT_VERSION = createHash("sha256")
  .update(EXTRACTION_PROMPT + JSON.stringify(SCHEMA))
  .digest("hex")
  .slice(0, 12)

Include the schema in the hash. A schema change alters behaviour as much as a wording change, and the two are usually edited together.

Decide the policy before shipping the change

Four options, and the right one depends on what the data is for:

Do nothing. New documents get the new prompt; old records stand. Correct when the change is an improvement rather than a correction — better handling of an edge case that was previously flagged for review rather than silently wrong.

Reprocess everything. Clean, expensive, and it rewrites history. If a customer has already reconciled a ledger against those records, changing them underneath is not obviously a favour.

Reprocess selectively. The usual answer. Identify records the change could plausibly affect and redo only those.

Reprocess on read. Leave the data, and re-extract lazily when a record is next accessed. Cheap and it means two records viewed on the same day can have been produced by prompts six months apart, which is difficult to reason about. Rarely worth it.

The decision belongs in the pull request that changes the prompt, alongside the eval numbers. Deciding afterwards means deciding under pressure, usually because a customer found one of the old errors.

Selective backfill, targeted properly

The whole saving is in not reprocessing 400,000 documents to fix 4,000. Narrow by what the change actually touched:

-- Candidates: extracted under the old prompt AND matching the profile
-- the change addressed. Here: a specific supplier's template.
select id from extractions
where prompt_version = 'a3f81c20b4de'
  and supplier_id = 'supplier-2281'
  and extracted_at >= '2026-05-01'
order by extracted_at desc;

Better still, target by symptom rather than by guess. If the change fixed a field that used to come back unreadable, the affected records are identifiable directly:

-- Records where the old prompt refused on a field the new one handles.
select e.id from extractions e
join extraction_fields f on f.extraction_id = e.id
where e.prompt_version = 'a3f81c20b4de'
  and f.field = 'vat'
  and f.status in ('unreadable', 'ambiguous');

This is the practical argument for refusal being a recorded state rather than a null. Refusals are a work queue for exactly this moment, and without them you are reprocessing on a hunch.

Never overwrite in place

Reprocessing should append, not replace. Two reasons, and the second is the one that bites.

You need to be able to compare — and a backfill that silently changed 4,000 records with no diff is impossible to audit. And a new prompt can be worse on specific documents even when its aggregate is better; without the old value you cannot see that happen.

create table extraction_versions (
  extraction_id  uuid not null,
  prompt_version text not null,
  fields         jsonb not null,
  extracted_at   timestamptz not null,
  is_current     boolean not null default false,
  primary key (extraction_id, prompt_version)
);

create unique index on extraction_versions (extraction_id)
  where is_current;

Then promotion is a deliberate flip, and it can be gated on the diff:

const diff = compareFields(oldVersion.fields, newVersion.fields)

if (diff.changed.length === 0) {
  // The change did not affect this document. Common, and worth counting —
  // a high no-change rate means the backfill was scoped too widely.
  return promote(newVersion)
}

// A changed value on a record downstream systems already consumed is
// not automatically an improvement. Sample these for a human.
if (diff.changed.some((f) => MATERIAL_FIELDS.includes(f))) {
  return queueForReview(extractionId, diff)
}

return promote(newVersion)

Downstream has already acted

The part that makes this a business problem rather than a data problem: those records were used. Invoices were paid, reports were sent, decisions were made.

Changing a total from €1,240 to €1,204 after payment is not a correction to a row — it is a discrepancy someone has to resolve with a supplier. So the pipeline needs to know whether a record has been consumed:

alter table extractions add column downstream_state text not null default 'new';
-- new | exported | reconciled | locked

Records in locked are not silently corrected. They generate an exception for a person, with both versions attached. This is the same principle as everywhere else on this site: the system should surface the discrepancy rather than quietly choose a side.

Reprocess as a batch

Backfills have no user waiting on them, which makes them the clearest possible case for the Batch API and its 50% discount. With a stable instruction prefix, caching stacks on top.

Tag the spend so it does not look like a production cost spike:

await usage.record({ feature: "invoice_extract", environment: "backfill", ... })

A 400,000-document reprocess arriving in the same bucket as live traffic is how a one-off backfill gets mistaken for a runaway feature, and cost attribution only helps if the backfill is labelled as one.

More in when extraction accuracy collapses, the same invoice, uploaded three times, and how we build extraction pipelines.