Fail open or fail closed — pick before you ship

The pipeline is unsure about a value. Two options:

Fail open. Write the record, mark it uncertain, let it flow downstream. The process keeps moving and a possibly-wrong value is now in the system.

Fail closed. Stop. Hold the record, put it in a queue, wait for a person. Nothing wrong enters the system and the process is blocked until somebody acts.

Most teams never make this decision explicitly. Which means it was made by whichever behaviour the framework defaulted to, and it is usually fail open, because writing a row is easier than building a queue.

Both choices are defensible and they have opposite failure modes. The problem is not picking wrong. The problem is not knowing which one you picked.

What each one costs

Fail open trades correctness for throughput. Its failure mode is silent corruption: bad values indistinguishable from good ones, discovered downstream — often by a customer. Its virtue is that nothing ever stalls.

Fail closed trades throughput for correctness. Its failure mode is a growing queue and a blocked process. Its virtue is that wrong data never enters the system of record.

The asymmetry that decides it: how expensive is a wrong value compared to a delayed one?

The pattern: fail closed where the output triggers an irreversible action, fail open where it feeds something statistical or easily corrected.

It is per field, not per pipeline

The useful refinement, and the one that makes this practical rather than a binary choice you regret.

A single invoice can do both. The supplier name at 0.99 confidence goes through. The total at 0.71 stops. There is no reason to hold nine good fields hostage to one uncertain one — that is a document-level decision applied to a field-level problem, and it is only forced on you if you have a single document-level confidence score.

type FieldPolicy = {
  /** Below this, the field does not pass. */
  threshold: number
  /** What happens when it does not. */
  onUncertain: "hold_document" | "hold_field" | "write_flagged"
}

const POLICY: Record<string, FieldPolicy> = {
  // Money moves. Nothing downstream sees this until a person confirms.
  total:     { threshold: 0.98, onUncertain: "hold_document" },

  // Wrong terms are recoverable, but the record should not post yet.
  issuedOn:  { threshold: 0.95, onUncertain: "hold_document" },

  // Useful, not critical. Write it, flag it, let reconciliation catch it.
  poNumber:  { threshold: 0.90, onUncertain: "write_flagged" },

  // Search only. Never blocks anything.
  description: { threshold: 0.0, onUncertain: "write_flagged" },
}

Writing this table is the exercise. It takes an hour, it requires talking to whoever owns the downstream process, and it converts an implicit default into a documented decision with a name against it.

Make the state visible in the data

A flagged value that looks like a confirmed value has failed open in the worst way — you took the throughput and did not get the audit trail:

alter table extracted_fields add column certainty text not null default 'confirmed';
-- confirmed | flagged | held

-- Downstream reads this view, not the table. Flagged values are visible
-- but cannot be mistaken for confirmed ones.
create view postable_fields as
  select * from extracted_fields where certainty = 'confirmed';

The view is the enforcement. If downstream systems read the raw table, every consumer has to remember to check certainty, and one that forgets reintroduces the bug for everyone. A view that only exposes confirmed values makes the safe path the default path.

Fail closed needs a release valve

The real objection to failing closed is that it can stop the business. If the queue grows faster than it drains, at some point somebody bulk-approves it to clear the backlog — and a human gate that gets bulk-approved is not a gate.

So a fail-closed policy needs an explicit escape, designed rather than improvised:

// When the queue exceeds what reviewers can clear, the correct response
// is a deliberate, recorded, time-boxed policy change — not a person
// clicking approve 300 times at 6pm.
if (queueDepth > CAPACITY_CEILING) {
  await raiseThresholds({
    reason: "queue_over_capacity",
    from: POLICY, to: RELAXED_POLICY,
    expiresAt: addHours(new Date(), 24),
    approvedBy: onCallLead,
  })
}

The properties that matter: it expires, it is attributed, and it is recorded. The same shape as bounding an approval queue — the system should degrade deliberately rather than have somebody degrade it quietly.

Where the default bites

Two places where fail-open happens without anyone choosing it:

A nullable column. If total is nullable, an extraction that produced nothing writes null, and a downstream report treats null as zero. That is failing open, decided by a schema.

A retry that gives up. Code that retries twice and then writes whatever it last got has failed open at the end of a retry loop, which is where nobody is looking.

Both are worth auditing for directly. The question to ask of any pipeline is simple: what does a value that we were not sure about look like in the database, and can anything downstream tell? If the answer is "the same as a good one" and "no", you are failing open regardless of what the design document says.

More in teach the extractor to refuse, the review queue is the product, and how we build extraction pipelines.