The agent sent the email twice

The agent sends an email. The SMTP call succeeds. Then the process is killed mid-write — a deploy, an OOM, a container rotation — before the row recording the send is committed.

The job is still on the queue, because the queue never saw an acknowledgement. It is picked up again. Everything downstream of "has this been sent?" reads the database, sees nothing, and sends it again.

The recipient gets the same cold email twice, four minutes apart. In a system whose entire premise is that it does not behave like spam, that is not a minor defect.

Every queue worth using delivers at least once. That is not a bug you can configure away — it is the guarantee. The duplicate has to be absorbed on your side.

Why "check before sending" does not work

The obvious fix is a read before the write:

// Broken. There is a window between the check and the send.
if (await alreadySent(recipientId, campaignId)) return
await sendEmail(...)
await markSent(recipientId, campaignId)

Two workers can both pass the check before either reaches the send. The window is small and it is real, and at any meaningful volume small windows are hit routinely. Worse, the failure above does not even need concurrency — the crash between sendEmail and markSent reproduces it with a single worker.

You cannot fix a distributed race with a read. You need the constraint enforced at the point of the write, by something that arbitrates.

Derive the key from the intent

An idempotency key identifies the action a caller intended, so that the same intent presented twice is executed once. Which means it has to be derived from the inputs, deterministically, before any side effect happens:

import { createHash } from "node:crypto"

function actionKey(a: {
  runId: string
  step: number
  kind: "send_email" | "create_invoice" | "post_message"
  target: string        // recipient, customer, channel
  payloadHash: string   // hash of the exact body being sent
}) {
  return createHash("sha256")
    .update([a.runId, a.step, a.kind, a.target, a.payloadHash].join(" "))
    .digest("hex")
}

The two common mistakes are both about what goes into that hash.

A random UUID generated at call time. A retry generates a new one, so the two attempts have different keys and both execute. This is the most frequent version of the bug, and it looks correct in review.

Including a timestamp. The same problem, dressed differently.

The key must be a pure function of the intent. Retry the same intent, get the same key.

Enforce it in the database

The check and the write have to be the same operation, which means a unique constraint:

create table action_log (
  key           text primary key,        -- the idempotency key
  run_id        text not null,
  kind          text not null,
  status        text not null,           -- in_flight | succeeded | failed
  result        jsonb,
  attempted_at  timestamptz not null default now(),
  completed_at  timestamptz
);

async function once<T>(key: string, meta: Meta, fn: () => Promise<T>): Promise<T> {
  // Claim the key. If another worker already holds it, this returns no rows.
  const claimed = await db.query(
    `insert into action_log (key, run_id, kind, status)
     values ($1, $2, $3, 'in_flight')
     on conflict (key) do nothing
     returning key`,
    [key, meta.runId, meta.kind],
  )

  if (claimed.rowCount === 0) {
    const prior = await db.one(`select * from action_log where key = $1`, [key])

    if (prior.status === "succeeded") return prior.result as T

    // in_flight means another worker is mid-execution, OR a previous worker
    // died holding the claim. Do not guess which. Throwing returns the job to
    // the queue; a reaper decides about stale claims separately.
    throw new InFlightError(key, prior.attempted_at)
  }

  try {
    const result = await fn()
    await db.query(
      `update action_log set status = 'succeeded', result = $2, completed_at = now()
       where key = $1`,
      [key, result],
    )
    return result
  } catch (err) {
    await db.query(
      `update action_log set status = 'failed', completed_at = now() where key = $1`,
      [key],
    )
    throw err
  }
}

The on conflict do nothing ... returning is doing the real work. Exactly one worker gets a row back; everyone else gets nothing and knows to stand down. The database arbitrates, because it is the only participant that can.

The stale claim is the part people skip

in_flight forever is what happens when a worker dies between claiming and completing. Left alone, that action never runs again — you have converted a double-send into a silent no-send, which is quieter and sometimes worse.

You need a reaper, and it needs a policy per action kind:

-- Candidates for recovery: claimed, never completed, older than the
-- longest plausible execution time for that kind of action.
select * from action_log
where status = 'in_flight'
  and attempted_at < now() - interval '15 minutes';

What the reaper does next is not a technical decision, it is a product one:

• Idempotent at the provider. The downstream API accepts its own idempotency key — Stripe does, and most payment and messaging APIs do. Retry freely; the provider absorbs the duplicate.
• Verifiable. You can query the provider to ask whether the action landed. Check, then decide. The best case where it is available.
• Neither. Plain SMTP is the honest example. You cannot ask a mail server whether it delivered a specific message. Here the choice is between possibly sending twice and possibly not sending at all, and for outbound email the correct answer is not to send. Mark it failed, surface it, let a person look.

That last case is why this is worth designing rather than solving with a retry decorator. The recovery policy encodes which error your business prefers, and for anything irreversible the answer is usually to stop and ask — the same argument as the approval gate.

Where to put the boundary

Wrap the smallest unit that has an external effect. One key per API call, not one per agent run.

An agent run that sends three emails and writes two records has five keys. If it dies after the fourth, resuming re-executes only the fifth — the first four claims are already succeeded and return their stored results. Key the whole run instead, and a crash near the end forces a choice between redoing everything and redoing nothing.

That granularity is also what makes a run genuinely resumable rather than merely restartable, which is a distinction worth its own article.

More in the approval gate is a state machine and agentic workflows with human approval gates.