Retry is not resume

A run of eleven steps fails at step nine. The job goes back on the queue, gets picked up, and starts at step one.

That is a retry. It is also, if any of the first eight steps touched the outside world, eight duplicate actions — and the framework doing it will report the eventual outcome as a success.

Retry re-executes. Resume continues. Most job runners give you the first and are described as if they gave you the second.

Where the conflation comes from

Retry is the correct primitive for a single idempotent operation. An HTTP GET that times out should be retried; that is what backoff libraries are for, and they are good at it.

The mistake is applying that primitive one level up, to a multi-step run. A run is not one operation. It is a sequence with accumulated state and side effects along the way, and re-executing it from the start is not a retry of anything — it is a second run that happens to share an id with the first.

// The shape almost every job runner encourages.
async function run(input: Input) {
  const data = await fetchRecords(input)      // step 1, harmless
  const scored = await scoreAll(data)         // step 2, costs money
  await writeScores(scored)                   // step 3, writes
  await notifyOwners(scored)                  // step 4, SENDS EMAIL
  await markComplete(input.id)                // step 5, fails here
}
// Retried: four steps re-execute. The owners are notified twice.

Step five failing is the most common shape, because the final bookkeeping write is usually the shortest and least defended step in the function.

Journal each step

Resume requires knowing what already happened, which means writing it down as it happens:

create table run_steps (
  run_id      text not null,
  step        int  not null,
  name        text not null,
  status      text not null,          -- succeeded | failed
  result      jsonb,                  -- the output, for downstream steps
  attempt     int  not null default 1,
  at          timestamptz not null default now(),
  primary key (run_id, step)
);

Then each step consults the journal before executing:

async function step<T>(
  runId: string,
  n: number,
  name: string,
  fn: () => Promise<T>,
): Promise<T> {
  const prior = await db.maybeOne(
    `select result from run_steps
     where run_id = $1 and step = $2 and status = 'succeeded'`,
    [runId, n],
  )
  // Already done on a previous attempt. Return what it produced;
  // do not do it again.
  if (prior) return prior.result as T

  const result = await fn()

  await db.query(
    `insert into run_steps (run_id, step, name, status, result)
     values ($1, $2, $3, 'succeeded', $4)
     on conflict (run_id, step) do update
       set status = 'succeeded', result = excluded.result, at = now()`,
    [runId, n, name, result],
  )
  return result
}

async function run(input: Input) {
  const id = input.runId
  const data   = await step(id, 1, "fetch",  () => fetchRecords(input))
  const scored = await step(id, 2, "score",  () => scoreAll(data))
  await step(id, 3, "write",  () => writeScores(scored))
  await step(id, 4, "notify", () => notifyOwners(scored))
  await step(id, 5, "close",  () => markComplete(id))
}

Now a failure at step five re-enters the function, skips four journal lookups that cost a millisecond each, and executes only what did not finish. No duplicate emails, and no re-paying for the scoring.

The gap the journal does not close

There is still a window: the side effect succeeds and the process dies before the journal row commits. On resume, the step looks unexecuted and runs again.

The journal narrows this from "the whole run re-executes" to "one step might re-execute", which is a large improvement and not a guarantee. Closing it properly needs the side effect and the journal write to be atomic:

• For database writes, put both in one transaction. Genuinely exactly-once.
• For external calls, use an idempotency key the provider honours, so a duplicate call is absorbed at the far end.
• For calls with neither, write the journal row as in_flight before executing. On resume, an in_flight row is a known-unknown: it is not retried automatically, it is surfaced to a person who can check.

That third case is not a workaround, it is the honest answer. Plain SMTP cannot tell you whether a message was delivered, and a system that pretends otherwise is choosing to double-send silently.

Steps have to be addressable

Journalling by ordinal position works until the code changes. Insert a new step three and every subsequent step shifts, so a resumed in-flight run maps its old step four onto the new step five and skips or repeats work.

Two constraints keep it sound:

Key by name, not just number. (run_id, name) survives reordering. Positional keys do not.

Pin the run to a code version. Record which version produced the journal. Resuming a run whose code has since changed is not obviously safe — the step boundaries may no longer mean the same thing. Fail it loudly and let a person decide rather than resuming into a different program.

When a retry is right

None of this argues against retries. It argues about the level they belong at.

Retry inside a step, for transient faults — a timeout, a 503, a rate limit. Backoff with jitter, bounded attempts. This is what retry is for.

Resume across steps, for anything that killed the process. Deploys, OOMs, node rotations, crashes.

The test for which you need is one question about each step: if this runs twice, does something happen twice? If the answer is no anywhere in the run, a plain retry is fine and the journal is overhead. The moment the answer is yes for even one step, you need resume, and the version where you keep the retry and add a comment saying "should be idempotent" is the one that sends the email twice.

More in the agent sent the email twice, the approval gate is a state machine, and agentic workflows.