Streaming costs the same and feels twice as fast

A four-second response delivered all at once and the same four-second response delivered token by token cost exactly the same. Same tokens, same model, same bill.

They are not the same product. One is a spinner for four seconds; the other starts answering in six hundred milliseconds and finishes reading at roughly the speed the user reads. The second feels perhaps twice as fast, and the difference is free.

Total latency is an engineering number. Time-to-first-token is the number a person experiences, and they are optimised by different things.

What streaming changes and does not

Does not change: token count, cost, total wall-clock to the final token, or model output.

Does change: when the user sees something. And past the first few hundred milliseconds, a response that is visibly progressing is tolerated for far longer than one that is not — the user is occupied reading rather than waiting.

There is a second, less-discussed benefit: streaming makes a wrong answer abandonable. A user who can see the answer heading somewhere unhelpful stops reading and rephrases at two seconds instead of waiting for the full four and then rephrasing. That is a real reduction in wasted calls.

Where time-to-first-token goes

The number worth instrumenting is not total latency — it is the gap before anything appears:

Everything above the last row is serial and blocks the user seeing anything. That is where the work goes, and note that the retrieval half is often a third of it — which is why gating the reranker is a user-experience change rather than a cost one.

Two things that quietly break streaming

Buffering in the stack. A proxy, a CDN, or a framework that collects the response before forwarding it converts a streaming endpoint back into a non-streaming one, silently. The endpoint still works; the benefit is gone.

Check for it directly rather than assuming:

# Timestamps on each chunk. If they all arrive together, something
# between you and the client is buffering.
curl -N -s https://api.example.com/chat -d '{"q":"..."}' \
  | while IFS= read -r line; do printf '%s %s\n' "$(date +%s.%N)" "$line"; done

The usual suspects are proxy_buffering in nginx, compression middleware that needs the full body, and serverless platforms that do not support streaming responses on the runtime you chose.

Waiting for structure. A response the client cannot render until it parses is not streaming from the user's point of view. If the model is producing JSON and the UI waits for a valid object, the user sees nothing until the last brace.

Where the output is structured and the surface is interactive, stream the prose and send the structure separately — or use a streaming-tolerant parser that can render partial objects. Otherwise be honest that this endpoint is not user-facing-fast and give it a progress indicator instead.

Stream progress, not just tokens

For anything with a retrieval or tool-use stage before generation, the first token is not the first thing you can send. Send the stages:

// The user sees motion during the 400ms before generation starts.
yield { type: "status", text: "Searching policy documents" }
const hits = await retrieve(query)

yield { type: "sources", items: hits.map(summarise) }   // useful and immediate
for await (const token of generate(query, hits)) {
  yield { type: "token", text: token }
}

Sending the sources before the answer is worth more than it looks. The user can see the system found the right documents while it is still composing, which is both reassuring and occasionally lets them correct course early.

For agent runs the same principle covers much longer waits: streaming "reading the invoice", "checking the supplier record", "drafting" turns a ninety-second opaque wait into something legible.

Cost is unchanged, with two footnotes

Streaming does not affect the bill. Two second-order effects are worth knowing:

Abandoned generations still cost. If the user navigates away, you are billed for tokens generated up to the point the connection closes — and only if you actually cancel. A server that keeps generating after the client disconnects pays for the whole response nobody will read:

// Propagate cancellation. Without this, an abandoned request runs to
// completion and is billed in full.
const controller = new AbortController()
req.signal.addEventListener("abort", () => controller.abort())
const stream = await client.messages.stream({ ...params }, { signal: controller.signal })

Streaming can reduce total spend. Users who abandon early cause fewer full generations, and fewer rephrase-and-retry cycles. Small, real, and in the opposite direction to what people expect.

When not to stream

Batch and asynchronous work. Nobody is watching. Use the batch API and take the 50% instead.

Structured extraction. The consumer is code. A partial JSON object has no value, and streaming adds parsing complexity for nothing.

Very short responses. A classification returning one word arrives in under a second either way. Streaming adds machinery to save nothing.

The rule: stream when a human is reading the output as it arrives. Otherwise it is complexity with no beneficiary.

More in where the milliseconds actually go, output tokens are five times the price, and what an AI feature costs to build.