Freeze the evidence, or you are not measuring the agent

Cloudflare published a long engineering post on 7 October 2026 describing Managed Defense, a security operations harness in which a coordinator agent runs four specialists in parallel — traffic analysis, customer context, global telemetry, threat intelligence — and a synthesis agent writes one advisory from their typed findings. It is by Deanna Tran and Blake Darché, and it is in early beta for eligible application-security alerts and cases.

The alert-triage content is not what I would take from it. What I would take is a design decision stated in two lines near the top, under a heading that reads "Recon first, inference second":

"It's tempting to put an AI agent at every step. The front half of our harness has none."

"Before we even call inference, deterministic code runs a fixed set of reconnaissance workflows with versioned API calls."

Most writing in this category describes the loop. This describes what happens before the loop — and the justification Cloudflare gives for it is not a security justification at all. It is an evaluation justification.

The front half of an agent is where your eval set lives, whether you designed it that way or not. If the agent fetches its own evidence, a score that moved cannot tell you whether the retrieval changed or the reasoning did.

The failure they name from their own prototype

The post is unusually direct about the version that came first. A single general-purpose agent was tried, and it produced three recurring problems, quoted here because the names are better than any I have come up with:

"Context became authority. A detection is a hypothesis, not proof that an exploit succeeded or an attack occurred. A broad AI agent can blur that distinction."

"Scope drifted. An AI agent can query the wrong account, time range, or source. You can't rely on a language model prompt to be a boundary."

"Failure disappeared. If a lookup times out, the result may not distinguish 'not checked' from 'checked and not found.'"

Two of those are ordinary agent problems. The third is the one that decides whether you can measure anything, and it is a schema bug, not a model bug. A lookup that never ran and a lookup that ran and found nothing are different states. If your tool layer returns null for both, they become the same state, and the second one is the answer that closes a case.

That is a design decision you make in about ten minutes and then live with for the life of the product.

What the frozen snapshot buys, in the post's own words

Cloudflare's recon half gathers its evidence up front, and the property they care about is stated plainly:

"Each piece of data is stored with its source, version, and timestamp."

"The fixed recon snapshot also makes evaluation reproducible."

with the consequence spelled out as differences between specialist agents' findings coming from "interpretation rather than retrieval."

If you run an agent product, that sentence is about you. The reason an agent score moves between two commits is one of two things: the evidence the agent retrieved changed, or the model's handling of the same evidence changed. With live fetching, both are true at once and you have no way to separate them. I have written about this diagnostic before as a post-hoc question — was that a retrieval failure or a generation failure. Moving it to design time is strictly better, because a versioned snapshot is cheap to build and retrofitting one onto a running agent is not.

The arithmetic: a stable prefix is also a cacheable prefix

There is a second consequence that the post does not mention, and it is the one that shows up on the invoice. A frozen snapshot is a stable prefix, and a stable prefix is exactly the precondition for prompt caching to do anything.

Take one investigation: four specialists fanning out in parallel, each needing the same recon snapshot. Give it a stated 18,000-token snapshot, 1,600 tokens of specialist instruction and query, and 900 tokens of output. At published Claude rates checked 7 October 2026 — Sonnet 5.5 at $2/MTok input, $10/MTok output, $2.50 cache write, $0.20 cache read:

Frozen snapshot, cache warm (case A)
  cached:    18,000 × $0.20/MTok = $0.003600
  uncached:   1,600 × $2.00/MTok = $0.003200
  output:       900 × $10/MTok   = $0.009000
  per specialist                  = $0.015800   × 4 = $0.063200

Live-fetched evidence, no stable prefix (case B)
  input:     19,600 × $2.00/MTok = $0.039200
  output:       900 × $10/MTok   = $0.009000
  per specialist                  = $0.048200   × 4 = $0.192800

case B ÷ case A = $0.1928 ÷ $0.0632 = 3.05×

Three times the specialist block, from one decision about where the evidence is fetched. There is a worse variant if the four specialists fire concurrently before any cache write lands — all four miss and all four write:

Concurrent cold fan-out (case C)
  write:   4 × 18,000 × $2.50/MTok = $0.180000
  input:   4 ×  1,600 × $2.00/MTok = $0.012800
  output:  4 ×    900 × $10/MTok   = $0.036000
  total                            = $0.228800   = 3.62× case A

$0.2288 against $0.1928 is 18.7% more than not caching at all — the same trap I worked through in fan-out loop order, where turning caching on for a concurrent fan-out makes the bill worse than leaving it off. Freezing the snapshot does not just make the eval reproducible; it is what makes the parallel fan-out cacheable in the first place.

At a stated 400 investigations a day surviving triage, that is $25.28/day under case A and $77.12/day under case B — a gap of $51.84 a day, or $1,555.20 a month on one design decision. The cheaper version is also the one you can measure.

A retry against live evidence is not the same experiment

The retry budget follows from the same fact. A failed investigation is normally retried, and the reason a retry is legitimate is that it re-runs the same work against the same inputs — retry is not resume, but retry is at least supposed to be a repeat. Against a live-fetched evidence set it is not. The second attempt reads a traffic baseline from four minutes later, a detection history that has grown, an enforcement outcome that may have changed.

Two things break at once. The retry costs the case-B number again, and it is no longer comparable to the attempt it is replacing — so a run that succeeds on the second try tells you nothing about whether the first failure was your code or the world. Flakiness that you cannot see is flakiness you will eventually attribute to the model, upgrade the model, and keep, which is how retries become a line item nobody can explain.

The three states, and the eval point they are worth

The fix Cloudflare describes for "failure disappeared" is a taxonomy rather than a prompt change. The advisory distinguishes three states:

"Not checked" · "Checked, with no matching result" · "Checked, with evidence supporting absence"

Those are three different findings and only the third one is evidence. The same distinction is drawn the same week by Claix's piece on retrieval for agents (5 October 2026), which lists "treating unknown as false" among the pitfalls of document pipelines. Two independent sources landing on the same clean-up in the same week is a reasonable signal that it is the one people skip.

Here is what it does to your measured accuracy. Suppose an investigation needs six lookups and each one completes 98% of the time. The chance that all six ran is 0.98 to the sixth:

0.98^6  = e^(6 × ln 0.98) = e^(6 × −0.02020271) = e^−0.12121625 = 0.88584
1 − 0.88584 = 0.11416  →  11.4% of investigations had at least one lookup
                            that never ran

Against a two-state schema, all 11.4% of those report the same thing a genuinely clean investigation reports. Your "no findings" rate has eleven points of blindness folded into it and looks healthier for it. The rate barely improves as the lookups get better, which is the part that surprises people:

"Basically always works" at the per-call level is one investigation in nine at the investigation level. You do not fix that with a better model. You fix it by having somewhere to put the answer.

Two more things worth taking

The citation check is application code. Cloudflare's validation step produces a versioned evidence package and then: "Application code checks that every citation exists, belongs to the investigation, and supports the attached claim." Note that it is not a model doing the checking, and note that the third clause is the hard one — existence and membership are set operations, but "supports the claim" is a judgement. Keeping that check in your own code, where it can fail loudly, is the same argument as citations that point at a passage rather than a document.

The cheap model is used at both ends. Clef, Cloudflare's decision model on Workers AI, does the high-volume triage on the way in, and then is used again at the end, where "Clef picks from a deterministically reduced list of attack classifications and dispositions." The expensive models only see the middle. That is a better cascade shape than the usual cheap-first-escalate one, because the small model also holds the acceptance gate — and constraining the output to a deterministically reduced set is a cost control as much as an accuracy control, given that output tokens are five times the price of input. The boundary around it is stated too: "The model never receives authority to cross tenant boundaries or act for the Managed Defense Analysts," scope confirmation is left to a human analyst, and "the Managed Defense Analyst remains responsible for the decision and any mitigation." That is an approval gate placed exactly where the blast radius is.

What I would not conclude from this

The post publishes no cost, latency, token, accuracy or volume figure. No number in this piece comes from Cloudflare — the 18,000-token snapshot, the six lookups and the 400-investigations-a-day volume are my stated scenario, and the rates are Anthropic's published ones, checked today. The three-state taxonomy and the prototype's failure modes are quoted from the post and are dated 7 October 2026.

I would also not read this as a claim that frozen evidence is always right. For an incident response system, evidence that is four minutes stale is not a reproducibility win, it is a wrong answer, and the recon/caching argument above is worth exactly nothing if your evidence is a moving target by nature. The decision is whether the thing you fetch during an investigation is a snapshot you can name or a live read. If it is a live read, you do not have an eval, you have a demo that has been run more than once.

For anything already in production, the honest first step is not to rebuild the front half — it is to find out how often "not checked" is currently being recorded as "found nothing," which is a query against data you already have. That is the kind of thing I look at in a reliability audit, and on a harness that is still being designed it is a schema and gate decision rather than a rewrite.

More in what an agent has to record so you can answer "why", an unattended run has no ceiling unless you build one, and where the milliseconds actually go. Quotations and the project's failure modes are from blog.cloudflare.com/agentic-security-operations, published 2026-10-07 by Deanna Tran and Blake Darché, read the same day. Model rates from claude.com/pricing, checked 7 October 2026 — Sonnet 5.5 at $2/$10 per MTok, which is now the current generation with Sonnet 5 listed as legacy at the same rates. The token counts, lookup count and investigation volume are a stated scenario, not measurements, and I ran none of this.