When the model bill triples and nobody knows why

The bill tripled. Usage did not. Nobody can say which feature is responsible, because every call goes out under one API key and comes back as one line on an invoice.

You cannot reduce a cost you cannot attribute. Instrumentation first, optimisation second.

Step 1 — make spend attributable

Before changing anything, tag every call with:

• feature — which product surface triggered it
• stage — which step of the pipeline
• model — exactly which version
• tokens in / out — recorded per call, not estimated
• cache hit — yes or no
• trigger — user action, batch job, retry, or eval run

An afternoon of work. It almost always identifies the problem immediately, because the answer is usually one specific thing rather than general growth.

The four we find most often:

1. A retry loop with no ceiling, quietly re-running failures
2. An eval suite in CI hitting the production key on every commit
3. One feature nobody thinks about — usually a background enrichment job — generating most of the volume
4. Full document context being sent when a page would do

Step 2 — the four things that dominate

Sending more context than the task needs

The most common and most expensive. A 40-page document sent whole when the answer is on page 3. Whole conversation histories replayed on every turn. Retrieval returning 20 chunks when 5 would do.

Input tokens are the bulk of most bills. Cutting context is the highest-return change available and usually improves accuracy too, because the model has less irrelevant material to be distracted by.

No caching

Two kinds, and most teams have neither.

Exact-result caching. Hash the input; if you have processed this page before, return the stored result. On real document workloads with boilerplate, repeated submissions and shared headers, this removes a large fraction of calls outright.

Provider prompt caching. If a long system prompt or document prefix is reused across calls, most providers will cache it at a substantially reduced rate. This requires structuring prompts so the stable part comes first — a refactor, not a config flag, but a cheap one.

One model for every task

Classification, routing, field extraction from clean text and complex reasoning do not need the same model. Route by difficulty:

• Small model handles everything first
• Escalate to the large model only when confidence is below threshold

This typically removes most large-model calls while improving accuracy, because building it forces you to have a confidence threshold at all.

Retries and agent loops without ceilings

An agent that loops is a billing incident with a stack trace. Enforce a per-run token ceiling and a per-day spend ceiling in the runtime, with a defined behaviour on hitting them that is not "crash silently and retry".

Step 3 — the ceiling nobody sets

Set a hard budget per run, per user, and per day, enforced in your code rather than hoped for. Providers give you an invoice, not a brake.

Alert on the rate of change, not the absolute number. A bill that doubles overnight is a bug. A bill that grows 15% a month alongside usage is a business.

What to expect

Ordered by return on effort:

The first row is not optional. Every team that skips it optimises the wrong thing.

Related: what an AI feature costs · the reliability audit