How many simulated conversations before the number means anything

Egma went up on Hacker News on 10 September 2026 — an open-source platform for testing voice agents with simulated conversations and watching them in production. The repository was created on 1 August, it is MIT licensed except for its ee folders, it supports LiveKit (JS/TS) and Retell, and it had 119 stars when I looked. The design is sensible in the ways that matter: tests live in your agent's repository, tool responses are mocked, and you bring your own provider keys so you pay for model usage directly.

One line in its feature list is where every team using a tool of this shape eventually gets hurt: "Set grading instructions and pass thresholds for the behavior you want to test."

A pass threshold is a number compared against a measurement. In a simulated conversation the measurement is produced by a model playing a caller, answered by a model playing your agent, and scored by a model playing a judge. None of those three is deterministic. The thing you are comparing against your threshold is not a measurement. It is a sample.

A simulated-conversation suite does not report your agent's pass rate. It reports one draw from a distribution whose mean is your agent's pass rate — and at the suite sizes people actually use, that draw is mostly noise.

A healthy suite, failing

Here is the whole problem in one number. Twenty scenarios. Each one genuinely passes nine times out of ten — a good agent, no regression, nothing wrong. Simulate a hundred thousand runs of that suite:

// evalnoise.ts — run with: node evalnoise.ts  (Node 23+ strips the types)

/** Deterministic PRNG, so this prints the same numbers for you as for me. */
function mulberry32(seed: number): () => number {
  let a = seed >>> 0
  return () => {
    a = (a + 0x6d2b79f5) >>> 0
    let t = Math.imul(a ^ (a >>> 15), 1 | a)
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296
  }
}
const rand = mulberry32(20260911)

/** One suite run: n scenarios, each an independent pass with probability p. */
function suite(n: number, p: number): number {
  let k = 0
  for (let i = 0; i < n; i++) if (rand() < p) k++
  return k
}

let allPass = 0, overThreshold = 0
for (let t = 0; t < 100_000; t++) {
  const k = suite(20, 0.9)
  if (k === 20) allPass++
  if (k >= 17) overThreshold++   // a 0.85 pass threshold
}

suite: 20 scenarios, true per-scenario pass rate 0.9, 100,000 runs
  all-must-pass gate is green in       12.3% of runs
  >=0.85 threshold gate is green in    86.7% of runs
  reported pass rate  p5 0.80  median 0.90  p95 1.00

An all-must-pass gate on a perfectly healthy suite is red seven times out of eight. That is not a hypothetical failure mode; it is the default configuration of every test runner anyone has ever used, because it is correct for unit tests, where a test that fails once fails always.

The threshold at 0.85 looks much better — until you notice what it is really saying. Thirteen per cent of the time, a build with nothing wrong with it is blocked.

What one green run is worth

The other direction is worse, because nobody investigates a green run. Suppose the suite comes back 18/20 and you write 0.90 in the release notes. The 95% Wilson interval around that observation:

95% Wilson interval for an observed 0.90, by suite size:
  n= 20  [0.699, 0.972]  width 0.273
  n= 50  [0.786, 0.957]  width 0.170
  n=100  [0.826, 0.945]  width 0.119
  n=200  [0.851, 0.934]  width 0.084
  n=500  [0.871, 0.923]  width 0.053

Twenty scenarios cannot tell 0.90 from 0.72. Your agent could be failing more than a quarter of real calls and the suite would hand you 18/20 without complaint. Not rarely — routinely.

Use Wilson, not the textbook normal approximation. Near a rate of 1.0 the normal interval runs off the end of the scale and reports an upper bound above 100%, which is exactly the region a passing eval lives in:

/** Wilson score interval — behaves near 0 and 1, where eval rates actually sit. */
function wilson(k: number, n: number, z = 1.96): [number, number] {
  const p = k / n
  const d = 1 + (z * z) / n
  const c = (p + (z * z) / (2 * n)) / d
  const m = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d
  return [c - m, c + m]
}

To pin the rate to within five points you need somewhere north of two hundred scenarios, and that is before anyone asks whether the simulated caller resembles a real one.

The same build, twice

Run the identical commit through the identical suite twice:

two runs of the SAME build (20 scenarios, p=0.9):
  second run scores lower              39.6%
  they differ by 2+ scenarios          41.4%
  the >=0.85 gate disagrees with itself 23.1%

Two in five times, the second run of an unchanged build scores lower than the first. Nearly a quarter of the time the gate flips. That last number is the mechanism behind the ritual every flaky test suite eventually acquires: it goes red, somebody reruns it, it goes green, everyone moves on. The ritual is locally rational — the rerun really does usually pass, because usually nothing was wrong. It is fatal anyway, because after a month nobody believes the suite, and the one real regression it catches gets rerun too.

What you can actually detect

The point of a suite is to notice change. So: you ship a genuine improvement, from a per-scenario pass rate of 0.85 to 0.92. How often does the suite tell you?

power to call a real 0.85 -> 0.92 improvement (one-sided, alpha 0.05):
  n= 20  17.6%
  n= 50  28.6%
  n=100  47.0%
  n=200  71.6%
  n=400  93.3%
  n=800  99.7%

Seven points of real improvement, and a twenty-scenario suite notices it about one time in six. Five times out of six the engineer who did the work sees a number that did not move, or moved the wrong way, and concludes the change was not worth it. The suite is not neutral here — it is actively steering the team away from improvements it cannot resolve.

Repeats are the lever

The instinct is to fix this by holding the scenario set fixed so the two runs are comparable. That helps, and it helps much less than I expected. With forty scenarios of genuinely unequal difficulty and a real improvement applied to all of them:

// paired.ts — same call budget, spent two ways.
const N = 40, MU = 1.8, SPREAD = 1.2, DELTA = 0.5
const sigmoid = (x: number) => 1 / (1 + Math.exp(-x))

/** Scenarios are not equally hard. A fixed suite means reusing these draws. */
const drawSuite = (n: number) => Array.from({ length: n }, () => normal() * SPREAD)

function runScenario(difficulty: number, shift: number, repeats: number): number {
  const p = sigmoid(MU + shift + difficulty)
  let k = 0
  for (let i = 0; i < repeats; i++) if (rand() < p) k++
  return k
}

/** Identical scenarios both sides, compared scenario by scenario. */
function pairedDetects(repeats: number): boolean {
  const s = drawSuite(N)
  const diffs = s.map(
    (d) => (runScenario(d, DELTA, repeats) - runScenario(d, 0, repeats)) / repeats,
  )
  const mean = diffs.reduce((a, x) => a + x, 0) / N
  const varr = diffs.reduce((a, x) => a + (x - mean) ** 2, 0) / (N - 1)
  const se = Math.sqrt(varr / N)
  return se > 0 && mean / se > 1.645
}

40 scenarios, unequal difficulty, a real improvement applied to all of them
power to detect it, one-sided, alpha 0.05:
  rewritten suite, 1 call per scenario   16.7%   80 calls
  fixed suite,     1 call per scenario   20.2%   80 calls
  rewritten suite, 5 calls per scenario  47.3%  400 calls
  fixed suite,     5 calls per scenario  52.3%  400 calls

Pairing on a fixed suite buys three to five points of power for free, so do it. But the lever that actually moves the number is repeats: same forty scenarios, five calls each instead of one, and power roughly triples. Which makes sense — one call per scenario measures each scenario with a single coin flip, and no amount of clever comparison recovers information that was never sampled.

Repeats buy a second thing that a bigger suite does not:

per-scenario pass rate over 5 repeats, one suite:
  {"always passes":21,"flaky":19,"always fails":0}
  a single run would have labelled every flaky one pass or fail at random

Nineteen of the forty scenarios pass sometimes and fail sometimes. Not one of them fails consistently. A single-run suite reports each of those nineteen as a clean pass or a clean fail, chosen at random, and a developer then goes and debugs whichever ones happened to land on fail. Those nineteen are the most valuable output of the whole exercise and single-run testing destroys the information before anyone sees it. Usually they are not agent bugs at all — they are scenarios where the grading instruction is ambiguous, which is a spec problem with a cheap fix.

Then price it

Repeats are not free, and Egma's bring-your-own-keys model makes that visible rather than hiding it in a subscription, which is the honest arrangement. A voice simulation burns the caller model, your agent's model, the grader, and — for real voice rather than text — speech in both directions.

Say a simulated call costs you $0.20 end to end. Measure yours; the number below is a placeholder, the arithmetic is not:

smoke suite   20 scenarios × 1 call  =    20 × $0.20 =  $4.00   per PR
powered run  200 scenarios × 5 calls = 1,000 × $0.20 = $200.00  per release

At thirty pull requests a month, $120 of smoke tests plus $800 for four release comparisons. That is the shape of the budget, and it points straight at the policy: the suite you run on every PR and the suite you make a decision with are not the same suite. The PR suite is a smoke test whose job is to catch things that broke completely, and it should be gated on a floor low enough that noise never trips it. The powered comparison runs when you are deciding something — a model swap, a prompt rewrite, a release — and it is allowed to cost two hundred dollars and take an hour, because the alternative is deciding from a coin flip.

What I would change on Monday

1. Stop using all-must-pass. On any suite scored by a model it is a random number generator. Gate on a rate.
2. Put the interval next to the rate. If your dashboard says 0.90, it should say 0.90 [0.70, 0.97] n=20 until somebody is embarrassed enough to grow n.
3. Repeat every scenario, and record the per-scenario rate. Three is enough to find the flaky ones; five is better. Anything that is not 0 or 1 goes on a list, and that list gets read before any agent code is touched.
4. Fix the scenario set and compare paired. Small win, no cost, and it stops suite edits from masquerading as agent changes.
5. Split the gate from the measurement. Cheap floor on every PR, powered comparison at decision points. Write the sample size into the runbook.
6. Mock everything you can. Egma mocking tool responses is variance reduction, and it is the right instinct — every source of randomness you remove from the harness is sample size you get back for free.

One last thing, aimed at where this whole category is heading. Egma's roadmap says it plans to use a production failure to propose a change to your agent, test it, and open a pull request. That is a good idea sitting on top of a dangerous one: an automatic fix loop validated by a noisy suite will confidently generate pull requests that fix nothing, because a 20-scenario suite will show a 7-point improvement that is not there about as often as it shows one that is. Whatever measures the fix has to be sharper than whatever proposes it.

The statistics here are not specific to voice, or to Egma. They apply to any suite where a model grades a model — which now includes most evals that gate a deploy. More in who owns the eval set and how to evaluate an LLM pipeline. If your agent's test suite is one of the ones being rerun until it goes green, that is exactly what an AI reliability audit is for, and it is the first thing I check before putting an agent near anything irreversible.

Egma's features, licence and platform support are from its README and the GitHub API, checked 11 September 2026. All output above was produced by the two scripts shown, run locally on Node 25 with a fixed seed of 20260911; the simulation counts are 100,000 runs for the gate figures and 20,000 for each power estimate.