Read the retry-after header

Exponential backoff with jitter is well understood and most retry libraries implement it correctly. The part that goes wrong is upstream of the algorithm: the provider usually tells you exactly how long to wait, and the code guesses instead.

// Standard, and wrong. The response carried the answer.
await sleep(2 ** attempt * 1000 + Math.random() * 1000)

A 429 response typically includes retry-after, and rate-limited APIs often include headers saying how many requests remain and when the window resets. Backing off exponentially from an arbitrary base either waits far longer than necessary — burning throughput you paid for — or far too little, producing another 429 and another retry.

Guessing the wait when the server sent you the number is not resilience engineering. It is ignoring a message.

Read the headers first

type RateLimitInfo = {
  retryAfterMs?: number
  resetAt?: Date
  remaining?: number
}

function readLimits(h: Headers): RateLimitInfo {
  const out: RateLimitInfo = {}

  const ra = h.get("retry-after")
  if (ra) {
    // Two legal forms: delta-seconds, or an HTTP date.
    const secs = Number(ra)
    out.retryAfterMs = Number.isFinite(secs)
      ? secs * 1000
      : Math.max(0, new Date(ra).getTime() - Date.now())
  }

  const reset = h.get("anthropic-ratelimit-requests-reset")
              ?? h.get("x-ratelimit-reset-requests")
  if (reset) out.resetAt = new Date(reset)

  const rem = h.get("anthropic-ratelimit-requests-remaining")
            ?? h.get("x-ratelimit-remaining-requests")
  if (rem) out.remaining = Number(rem)

  return out
}

async function waitFor(info: RateLimitInfo, attempt: number) {
  // The server's number wins, when it gave one.
  if (info.retryAfterMs !== undefined) {
    // Small jitter even here: many workers get the same retry-after and
    // would otherwise resume in lockstep.
    return sleep(info.retryAfterMs + Math.random() * 500)
  }
  if (info.resetAt) {
    return sleep(Math.max(0, info.resetAt.getTime() - Date.now()) + Math.random() * 500)
  }
  // Only now guess. Full jitter: uniform in [0, cap], not cap ± noise.
  const cap = Math.min(60_000, 500 * 2 ** attempt)
  return sleep(Math.random() * cap)
}

Two details in that fallback are worth stating.

Full jitter, not additive jitter. random() * cap spreads retries uniformly across the window. cap + random()*1000 clusters them at roughly the same moment, which is a thundering herd with a decorative wobble on it.

Jitter even on retry-after. If forty workers hit the limit together, they all receive the same retry-after and, without jitter, all resume in the same millisecond — reproducing the burst that caused the limit.

Slow down before you are told to

remaining lets you avoid the 429 rather than recover from it. If a worker sees its remaining request budget getting low, throttling proactively is strictly better than sprinting into the wall:

// After every successful call, adapt. Cheap, and it smooths the whole fleet.
if (info.remaining !== undefined && info.resetAt) {
  const msLeft = info.resetAt.getTime() - Date.now()
  if (info.remaining > 0 && msLeft > 0) {
    // Space remaining requests evenly across the remaining window.
    const spacing = msLeft / info.remaining
    if (spacing > 50) await sleep(spacing)
  }
}

This converts a burst-then-block pattern into a steady rate, which finishes the same work sooner because none of it is spent waiting out a penalty window.

Not everything should be retried

A retry loop that retries the wrong things turns a fast failure into a slow one, and pays for each attempt:

The timeout row is the dangerous one. A timeout means you do not know whether the call happened, which is a different situation from a clean 500. Retrying a timed- out request that actually succeeded is how an agent sends the same email twice — so a timeout retry is only safe behind an idempotency key.

Bound the total, not just the attempts

maxAttempts: 5 is not a bound on anything a user cares about. Five attempts with exponential backoff can take several minutes, and if the caller gave up ninety seconds ago the last three attempts cost money for a result nobody will read.

Bound wall-clock instead:

async function withRetry<T>(fn: () => Promise<T>, opts: {
  deadline: Date            // absolute, propagated from the caller
  maxAttempts?: number
}): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn()
    } catch (err) {
      if (!isRetryable(err)) throw err
      if (attempt + 1 >= (opts.maxAttempts ?? 5)) throw err

      const wait = await computeWait(err, attempt)
      // Do not sleep past the deadline just to make one doomed attempt.
      if (Date.now() + wait > opts.deadline.getTime()) {
        throw new DeadlineExceeded(err)
      }
      await sleep(wait)
    }
  }
}

Passing a deadline down rather than a timeout at each layer is what stops nested retries from multiplying — three layers each retrying five times is 125 attempts, and every one of them is billed.

Retries are not free

Each retry of a failed model call costs input tokens again. On long prompts that is real money, and it is money that shows up on no dashboard unless you attribute it:

await usage.record({
  feature, runId,
  attempt,                              // > 1 means this was a retry
  outcome: attempt > 1 ? "retry" : "ok",
  costUsd: costFor(res.usage, model),
})

Without that column, a rate-limit incident looks like a general cost increase and the retry storm that caused it is invisible.

More in the agent sent the email twice, retry is not resume, and agentic workflows.