You are paying for every failed call

A document fails schema validation. The pipeline retries. It fails again, retries again, and succeeds on the third attempt.

Reported: one document processed successfully. Billed: three calls. The two failures cost exactly what the success cost, produced nothing, and appear nowhere except in the total on the invoice.

Retries are the only significant cost in an AI pipeline that is invisible by default. Everything else has a name attached to it.

Why they hide

Retry logic lives inside a wrapper, a library, or an SDK. From the caller's perspective one function was called and one result came back — which is the whole point of a retry abstraction, and also why the accounting disappears.

The provider knows there were three calls but not that two were retries of the first. You know they were retries and, unless you wrote it down, do not record it. Neither side has the complete picture.

The fix is one column, and it has to be at the innermost layer where the actual HTTP call happens:

// Inside the wrapper, per ATTEMPT. Not per logical operation.
await usage.record({
  feature,
  runId,
  model,
  attempt,                                  // 1, 2, 3
  outcome: ok ? (attempt === 1 ? "ok" : "ok_after_retry")
              : classify(error),            // rate_limited | schema_invalid
                                            // | timeout | server_error
  inputTokens, outputTokens,
  costUsd: costFor(usage, model),
})

Then the number exists:

select
  feature,
  sum(cost_usd)                                    as total,
  sum(cost_usd) filter (where attempt > 1)         as retry_spend,
  round(100 * sum(cost_usd) filter (where attempt > 1) / sum(cost_usd), 1)
                                                   as retry_pct
from model_usage
where environment = 'prod' and at > now() - interval '30 days'
group by 1
order by retry_pct desc;

The first time a team runs that query the number is usually somewhere between 8% and 25%. It is occasionally much higher, and it is never zero.

The four causes, and what each one means

Schema validation failures. The model returned something that did not fit, so you retried. This is the most fixable of the four — and the fix is usually not a better prompt but native structured output, which makes the shape a constraint rather than a request. A high schema-retry rate on one feature is a design signal, not a tuning problem.

Rate limits. Retries after a 429 that was avoidable. If your workers sprint into the limit and back off, you are paying for the collisions. Reading the rate limit headers and pacing proactively removes most of this.

Timeouts. The expensive category, because a timed-out call may have completed server-side. You are billed for the first attempt regardless, and the retry is a second full charge for the same work. Long max_tokens and long prompts make these more likely; both are worth tightening.

Genuine server errors. 500s and 529s. Not your fault, not avoidable, and correctly retried. This is the floor — a small percentage that should stay small.

The categories matter because three of the four are fixable and one is not. Reporting them as one "retry" bucket makes the whole thing look like weather.

Retries multiply through layers

The version that produces a shocking number is nested retry logic. An SDK retries three times. Your wrapper retries three times around the SDK. A job queue retries the whole job three times.

That is 27 attempts for one logical operation, and each one is billed. Nobody designed it — each layer was added by someone who did not know about the others, and each is individually reasonable.

Two rules prevent it:

Retry at exactly one layer. Usually the innermost one that can see the error type. Turn off SDK-level retries if your wrapper handles them, or the reverse — but not both.

Propagate a deadline, not a timeout. An absolute deadline passed down through the stack means inner layers stop when the outer one has given up, instead of each layer independently spending its full budget.

Audit it directly rather than by reading code:

-- Attempts per logical operation. Anything above about 4 means
-- more than one layer is retrying.
select max_attempt, count(*) from (
  select run_id, step, max(attempt) as max_attempt
  from model_usage group by 1, 2
) t group by 1 order by 1 desc;

A long tail out to 9 or 27 is the signature, and the powers-of-three pattern tells you how many layers are involved.

Retries that were never going to work

The worst spend is retrying something deterministic. A 400 from a malformed request will be malformed on the next attempt too. A document the model cannot read will not become readable.

The second case is subtle because it looks transient — the call succeeds, the output just fails validation again. A retry budget on content failures, separate from transport failures, stops this:

// Transport failures: retry per backoff policy.
// Content failures: one retry, then a human. The model is not going to
// suddenly read the smudged total on attempt three.
if (kind === "schema_invalid") {
  if (attempt >= 2) return review(doc, "extraction failed twice")
}

One retry on content is worth having — sampling variation genuinely does rescue some cases. The third and fourth attempts are close to pure waste, and they delay the review that was always going to be needed.

What to do with the number

Once retry spend is visible, it becomes an ordinary optimisation target and usually a well-ranked one:

1. Sort features by retry_pct. The worst one is almost always a schema problem with an obvious fix.
2. Alert on the trend, not the level. A feature going from 9% to 20% has a new failure mode, and that is worth knowing before the invoice.
3. Include it in unit economics. Cost per document computed from successful calls only is understated by whatever the retry rate is. Quote the number that includes them, or the margin on a fixed-price contract is wrong from day one.

More in which feature tripled the bill, when the model bill triples, and what an AI feature costs to build.