Every agent needs a dry run

How do you test a change to an agent that sends email?

The honest answers people give: point it at a test inbox, or run it and watch carefully, or ship it on Friday afternoon and check the sent folder. All three are testing in production with extra steps, and none of them scale to running the change against three hundred historical cases.

An agent you cannot run without consequences is an agent you cannot test. Dry run is not a debugging convenience — it is what makes the thing verifiable at all.

Not a flag on the tool

The tempting implementation is a boolean checked inside each tool:

// Fragile. Every tool must remember, forever, including the one
// added next quarter by someone who has not read this file.
async function sendEmail(args: EmailArgs, ctx: Context) {
  if (ctx.dryRun) {
    log.info("would send", args)
    return { ok: true }
  }
  return await smtp.send(args)
}

One tool that forgets the check sends real email during a dry run, and the failure is discovered by the recipient. The check is also invisible at the boundary where it matters — nothing structurally prevents a tool from having side effects.

Put it at the dispatcher

Make it a property of the execution environment, and classify tools rather than trusting them:

type ToolKind = "read" | "write" | "external"

type Tool = {
  name: string
  kind: ToolKind
  schema: JSONSchema
  execute: (args: unknown) => Promise<unknown>
  /** Required for write and external. What a dry run returns instead. */
  simulate?: (args: unknown) => Promise<unknown>
}

async function dispatch(tool: Tool, args: unknown, ctx: Context) {
  if (!ctx.dryRun || tool.kind === "read") {
    return await tool.execute(args)
  }

  // A side-effecting tool with no simulator cannot run in dry mode.
  // Fail loudly at registration time, not at call time.
  if (!tool.simulate) {
    throw new Error(`${tool.name} has no simulate() and cannot dry-run`)
  }

  await ctx.transcript.append({ kind: "simulated_call", tool: tool.name, args })
  return await tool.simulate(args)
}

Now a new tool cannot be registered without declaring its kind, and a side-effecting tool without a simulator fails at startup rather than in a run. The safety property is enforced by the dispatcher, which is the only place it can be enforced.

Simulate realistically, including failure

A simulator that always returns success produces a dry run that only exercises the happy path — which is the path least in need of testing:

const sendEmail: Tool = {
  name: "send_email",
  kind: "external",
  execute: (a) => smtp.send(a),

  simulate: async (a) => {
    // Validate exactly as the real path does. Most bugs are here,
    // and a simulator that skips validation misses all of them.
    const parsed = EmailArgs.parse(a)
    await checkSuppressionList(parsed.to)
    await checkDailySendCap(parsed.from)

    return {
      messageId: `dry-${hash(parsed)}`,
      accepted: [parsed.to],
      simulated: true,      // present in the result, so downstream can tell
    }
  },
}

Two properties worth insisting on.

Deterministic ids. dry-${hash(args)} rather than a random id, so the same inputs produce the same run and two dry runs can be diffed.

simulated: true in the payload. If a simulated result is ever written somewhere real, it is identifiable. Cheap insurance against the failure mode where dry-run output leaks into production data.

Where you can afford it, simulate stochastically too: return the provider's real error shapes some fraction of the time so retry and error-handling paths are exercised. A dry run in which nothing ever fails does not test the code that matters most.

What it lets you do

Regression-test against history. Replay three hundred past runs against a new prompt and diff the decisions. This is the payoff, and it needs the transcript to have recorded the inputs:

const changed = []
for (const run of await transcript.sample(300)) {
  const replay = await execute(run.inputs, { dryRun: true, promptVersion: "v9" })
  const diff = compareDecisions(run.decisions, replay.decisions)
  if (diff.length) changed.push({ runId: run.id, diff })
}
// "This prompt changes behaviour on 14 of 300 past runs, here they are."

That sentence is what turns a prompt change from a judgement call into a reviewable diff.

Preview before approval. A pending approval can show the reviewer the full simulated consequence — every email, every write — rather than a summary of intent.

Estimate cost. A dry run over a batch reports what the real run would spend, before committing to it.

The honest limits

Dry run is a simulation, and simulations diverge:

State that the real run would have changed. A run that writes a record and then reads it back gets the real value in production and a simulated one in dry run. Divergence compounds from there.

Timing. Real API latency, rate limits and retry behaviour are absent.

The world moving. A dry run against last month's data is a dry run against last month.

Which is why it does not replace a staging environment with real side effects against non-production systems. It replaces the much larger category of testing that would otherwise not happen at all, because the only alternative was sending real email.

More in if you cannot replay it, allow-lists, not deny-lists, and agentic workflows with human approval gates.