The same 2,500 calls, two loop orders, half the bill

On 10 September 2026 a research post from Scaled Cognition called Prompt Trees went up on Hacker News. It had been published the previous December, which is why I had not read it. It is about training, and I do not train models, so I nearly skipped it.

The result it reports is that branching data does not have to be flattened. If you have one system prompt with fifty conversations under it, or one prompt with many sampled rollouts — the shape of every RL step — the standard approach writes out each root-to-leaf path as its own sequence and re-encodes the shared part once per path. A prompt tree keeps the branching, and uses a block-sparse attention mask built from a per-token path vector, packed as bits into int64 chunks, to reproduce the same gradients while encoding each token once. Trained on a cluster of Nvidia H200 GPUs, they report "a ~70x speedup when the cached token ratio is ~100", and — the number I trust more — "a 3% penalty induced by the prompt tree attention implementation" when there is nothing shared to cache.

The 70x is not available to me at inference. But the metric they measure it against is, and it is the sharpest version of a thing I keep finding in production bills.

The ratio they use

Their x-axis is "the ratio of cached tokens to real tokens in the prompt tree" — 1 when every token is unique. Turn that around and you get the number I actually want:

tokens sent  ÷  tokens that carry information

If you send six times more tokens than your data contains, you are paying six times for the same information, and nothing in your dashboard will say so. Every production workload that fans out from a shared context has this ratio. An eval suite is the clearest example: N prompt candidates, each run against the same M cases, where the case itself is the bulk of the prompt.

The ratio is not a caching question. It is a layout question, and you answer it before you type a single cache_control block.

The same calls, two loop orders

Here is a shape I have seen built both ways, and the two versions differ only in which loop is on the outside.

500 eval cases × 5 prompt candidates.

  each case carries 6,000 tokens of context — identical across the five
  candidates, because it is the case
  each candidate carries 1,200 tokens of instructions and rubric —
  identical across the 500 cases, because it is the candidate

  output tokens are the same in every arrangement and are left out.

How much of the traffic is information?
  unique =   500 × 6,000   +   5 × 1,200  =  3,006,000
  sent   =   500 × 5 × (6,000 + 1,200)    = 18,000,000
  ratio                                        5.99

Six times. Now the two arrangements, at published Claude rates checked 13 September 2026 — Sonnet 5 at $2/MTok input, 5-minute cache write $2.50, cache read $0.20.

A — candidate outer, case inner. This is the natural way to write it, because you iterate the thing you are comparing:

for each candidate: for each case

  the candidate's 1,200-token instruction is the shared prefix
    write          1,200 × $2.50   = $0.0030
    read      499 × 1,200 × $0.20  = $0.1198
    cases     500 × 6,000 × $2.00  = $6.0000
                                     -------
                                     $6.1228   × 5 candidates = $30.61

B — case outer, candidate inner. Transposed:

for each case: for each candidate

  the case's 6,000-token context is the shared prefix
    write          6,000 × $2.50   = $0.0150
    read        4 × 6,000 × $0.20  = $0.0048
    instruct    5 × 1,200 × $2.00  = $0.0120
                                     -------
                                     $0.0318   × 500 cases = $15.90

uncached   18,000,000 × $2.00  = $36.00
A                                $30.61    15% off
B                                $15.90    56% off

Same model, same 2,500 calls, same prompts, same answers. The only difference is which for is on the outside, and it is worth 1.9×.

Why order is the whole decision

A prefix cache is keyed on a prefix. You get exactly one per call, and everything after the breakpoint is billed at base input — so the decision is which of the two things that repeat goes in front, and the answer is the one carrying the most repeated token mass.

In A, the instruction repeats 500 times: 1,200 × 500 = 600,000 tokens of repeated mass, and caching captures $0.123 of it while 15,000,000 tokens of context go by at full price. In B, the context repeats five times per case and 15,000,000 tokens of mass end up behind the cache.

There is a trap in the ratio worth naming, because I fell into it. In B the instruction is sent 2,500 times and is only 5 things — a redundancy of 500×, the highest in the job. It is still $6.00 of the $15.90, because it is billed at base every time. Redundancy tells you how many times something is duplicated; it does not tell you what that duplication costs. Weight by token mass, not by how silly the ratio looks.

The failure mode is a stampede

B has an assumption in it that is not in the arithmetic: that the four sibling calls read a cache the first call already wrote. If you dispatch the five candidates for a case concurrently and all five arrive before that write lands, you get five writes and no reads:

per case   5 × 6,000 × $2.50  = $0.0750
           5 × 1,200 × $2.00  = $0.0120
                                -------
                                $0.0870  × 500 = $43.50

$43.50. That is 21% more than not caching at all, and 2.7× the $15.90 the restructure was supposed to cost. The code is identical. Only the concurrency changed.

Whether a sibling can read a write still in flight is a property of your dispatch and the endpoint, not of your source, so measure it rather than assuming. The usage object reports both counts, and one query tells you which world you are in:

-- Per fan-out: one write and four reads is the design.
-- Five writes is a stampede; zero reads means you paid the premium for nothing.
select case_id,
       count(*) filter (where cache_creation_input_tokens > 0) as writes,
       count(*) filter (where cache_read_input_tokens     > 0) as reads,
       count(*)                                               as calls
from model_usage
where run_id = $1
group by 1
order by writes desc
limit 20;

If it is stampeding, the fix is not a bigger cache — it is making the first call of each fan-out complete before its siblings start, which costs you one round trip per case and buys back $27.60 per run.

What I am not taking from the paper

The 3% penalty is the honest part of the source and the reason I read the rest of it. The tree machinery is not free when there is nothing shared — which, at inference, means a transposed loop over a workload with no common prefix is all cost and no benefit. Compute the ratio first; if it is near 1, this is not your problem and you should go and find out which feature actually moved the bill.

Two other caveats in the post transfer as warnings rather than advice. They state plainly that they "do not speed up the time for on-policy sampling, which is another large part of that inner loop" — fixing the gradient path did not fix the generation path. The same is true here: transposing a loop changes what you are billed and not how long anything takes. If latency is the problem, this is the wrong lever, and the right one is batching what is not waiting on you.

And their note on loss weighting is the one I would carry into any fine-tuning I did do. With loss computed over internal nodes, a shared node "will only have 1/20 the weight in the optimization with prompt tree training" — which is the training-side version of the same bug: a prefix shared by twenty examples is twenty times the data, and if you do not decide that deliberately, you have decided it accidentally.

What I would do before touching a loop

Compute the ratio. Total input tokens sent, divided by the tokens that are actually distinct. I want one query per workload that reports this weekly. Near 1 means stop. Between 3 and 6 means there is money here.

Order the loops so the biggest repeated mass is the prefix. Not the highest redundancy ratio — the biggest mass. Those are different things and the second one is the one that pays.

Verify with the counters, not the config. Exactly one write per fan-out, or you have built the $43.50 version and the code looks correct in review.

Keep the outer key stable. The moment something per-case and variable lands in the prefix — a timestamp, a run id, a shuffled case order — the hit rate is zero and all of the above is decoration. It is the same discipline as ordering a single prompt correctly, applied one level up.

Check it holds after a provider change. The loop order is worth nothing on an endpoint that does not honour the cache the way you assumed, which is the trap in walking a failover chain mid-run.

This is the sort of thing that shows up when a pipeline gets an extraction and reprocessing pass and nobody has re-priced it since — and it is the kind of change I would rather make deliberately than discover on an invoice. More in what an AI feature costs to build and when the model bill triples.

Prompt Trees was published by Scaled Cognition on 01 December 2025 and surfaced on Hacker News on 10 September 2026; all quoted figures and the H200 detail are theirs, read from the post on 13 September 2026. Model rates are from claude.com/pricing, checked 13 September 2026. The 500 × 5 shape, the 6,000 and 1,200 token counts and the output-is-excluded assumption are my stated scenario, not measurements — substitute your own numbers and the ratios hold, the dollars move.