Paying to re-read the same tokens

A chat feature appends each turn to the history and sends the whole thing back. It is the simplest correct implementation and every framework's default.

It also means turn twenty re-reads and re-pays for turns one through nineteen. Not once — every turn re-reads everything before it, so the total cost of a conversation grows with the square of its length.

Nobody budgets for a chat feature by squaring the turn count, and that is exactly how the bill behaves.

The arithmetic

Assume 500 tokens added per turn — a user message plus a model reply — on Sonnet 5 at $2/MTok input.

Turn 40 costs forty times turn 1, and the conversation to that point has cost $0.82 in input tokens alone. Double the length and the total roughly quadruples.

Add a 20,000-token system prompt with retrieved context and the base is higher on every turn, which is where a support assistant's bill actually comes from — not from the conversations, from re-reading the same policy documents four hundred thousand times.

Cache the prefix first

Before pruning anything, take the discount that requires no product decisions. The stable part — system prompt, tools, retrieved documents — is identical every turn, which is exactly what prompt caching is for. A cache read is a tenth of base input.

Two rules make it work in a conversation:

Everything stable goes first. Any per-turn value in the prefix — a timestamp, a turn counter — invalidates the cache on every request.

Move the breakpoint as the conversation grows. Mark the cache boundary after the settled part of the transcript, so earlier turns are read at cache rates and only the newest turns are charged at base input.

That alone converts most of the quadratic growth into something much flatter, without changing behaviour at all.

Then prune, in one of three ways

Sliding window. Keep the last n turns and the system prompt. Trivial, and it loses the earlier conversation completely — the user says "as I mentioned" and the model has no idea.

Summarise and compact. When the transcript passes a threshold, replace the oldest turns with a summary:

async function compact(history: Turn[], keepRecent = 6): Promise<Turn[]> {
  if (history.length <= keepRecent + 4) return history

  const old = history.slice(0, -keepRecent)
  const recent = history.slice(-keepRecent)

  const summary = await summarise(old, {
    // Being specific about what to preserve is the whole job. A generic
    // "summarise this" loses the facts and keeps the pleasantries.
    preserve: [
      "decisions made and commitments given",
      "identifiers: order numbers, account ids, dates",
      "user-stated constraints and preferences",
      "anything the user corrected the assistant about",
    ],
  })

  return [{ role: "system", content: `Earlier conversation:\n${summary}` }, ...recent]
}

The last preserve item earns its place. A correction the user made — "no, it's the Manchester office" — is the single worst thing to lose, because the model will confidently repeat the original error and the user will have to correct it twice.

Note the trade: compaction costs a model call, so it only pays when the conversation continues well past the compaction point. Compacting at turn eight in conversations that average nine turns loses money.

Retrieve from the transcript. For long-running sessions, store turns and retrieve the relevant ones instead of carrying everything. This is RAG over the conversation, with the same properties and the same failure modes.

Where to set the threshold

Not by token count alone. By where your conversations actually end:

select percentile_cont(0.5)  within group (order by turns) as p50,
       percentile_cont(0.9)  within group (order by turns) as p90,
       percentile_cont(0.99) within group (order by turns) as p99
from conversations
where started_at > now() - interval '30 days';

If p90 is eight turns, a compaction threshold of twenty affects almost nothing and adds a code path that is rarely exercised — which is worse than not having it, because rarely-exercised paths break silently. Set it just above p90 so it runs often enough to be trusted, or do not build it.

The tool-result variant

Agent runs have the same shape and worse constants. Tool results are frequently large — a file read, a query result, a page of HTML — and every one of them is re-sent on every subsequent turn.

An agent that reads five 8,000-token files has 40,000 tokens of file content in context, re-read on every turn thereafter. That is the mechanism behind loops getting more expensive the longer they run.

The fix is the same shape, applied to results rather than messages:

// Keep the most recent tool results in full. Replace older ones with
// a reference the agent can re-fetch if it genuinely needs them again.
function truncateOldResults(history: Turn[], keepFull = 3): Turn[] {
  const toolResults = history.filter((t) => t.kind === "tool_result")
  const cutoff = toolResults.length - keepFull

  return history.map((t, i) =>
    t.kind === "tool_result" && i < cutoff
      ? { ...t, content: `[${t.tool} result, ${t.tokens} tokens, ref: ${t.id}]` }
      : t,
  )
}

Giving the agent a recall_result(id) tool alongside this makes the truncation recoverable rather than lossy — it re-reads only what it actually needs, which is usually one of the five files rather than all of them.

What not to do

Do not truncate the system prompt. It is the cached part. Cutting it saves tokens priced at 10% and risks the instructions that keep the model on task.

Do not drop the first user turn. It usually contains the actual request. Sliding windows that drop it produce an assistant that has forgotten what it was asked.

Do not compact mid-task. Summarising while an agent is halfway through a multi-step operation loses the state it was carrying. Compact at turn boundaries where nothing is in flight.

More in when the model bill triples, prompt caching with the arithmetic, and output tokens are five times the price.