Where the milliseconds actually go

A RAG endpoint feels slow. The team tunes the vector index — HNSW parameters, smaller dimensions, a faster instance.

Vector search was 12 milliseconds. It was never the problem, and it was the only stage anybody had a dashboard for.

Every stage of a RAG request is instrumented except the ones that take the time, because the slow ones are network calls that look like a single line of code.

Where it actually goes

A measured breakdown of a fairly ordinary retrieval-augmented answer:

Two things fall out immediately. Vector search is a rounding error. And the retrieval half — the part you control completely — is a meaningful fraction of time-to-first-token, which is the number users actually feel.

The three worth attacking

Query embedding. An API call on the critical path, and it usually runs before everything else because the vector is needed for the search. That makes it pure serial latency.

Two fixes. Cache embeddings for repeated queries — in any real product the head of the query distribution is heavy, and a cache hit removes the stage entirely. And run a local embedding model for queries if latency matters more than the last few points of retrieval quality; query embedding is far less sensitive to model quality than document embedding, because queries are short and the comparison is against your own indexed vectors.

Serial stages that could be parallel. This is the free one:

// Serial: embed, then search, then fetch. ~200ms.
const vector = await embed(query)
const hits = await vectorSearch(vector)
const payloads = await fetchChunks(hits.map((h) => h.id))

// Parallel where the dependency graph allows. ~120ms.
const [vector, lexical] = await Promise.all([
  embed(query),
  bm25Search(query),          // needs no vector — start it immediately
])
const dense = await vectorSearch(vector)
const fused = rrf([dense, lexical])
const payloads = await fetchChunks(fused.slice(0, 10))

Lexical search does not need the embedding. Running it concurrently makes it free — its entire duration hides inside the embedding call. Teams routinely run it after, serially, for no reason other than that the code reads top to bottom.

The N+1 on payloads. Vector stores return ids and scores. Fetching the text one row at a time is fifty round trips:

// One query. Obvious, and routinely not done because the ORM makes
// the per-id version look natural.
const rows = await db.query(
  `select id, content, metadata from chunks where id = any($1)`,
  [ids],
)

Gate the reranker

Reranking is the largest controllable cost and it is not needed on every query. When the first-stage results are already decisive, skipping it removes 200ms for free:

// Scores bunched together means genuine ambiguity — rerank.
// A dominant top hit means the ordering is already confident — skip.
const margin = hits[0].score - hits[2].score
const ranked = margin < 0.05 ? await rerank(query, hits) : hits

On typical traffic this skips a third to a half of queries. The p50 improves substantially; the p99 does not, which is the correct outcome — the hard queries still get the full treatment.

Overlap retrieval with generation

The trick that removes retrieval from the perceived critical path entirely: start streaming before retrieval finishes.

For a two-stage answer — a short acknowledgement or plan, then the substantive answer — the first stage needs no retrieved context. Begin it immediately, and splice the retrieved chunks in for the second stage.

This is genuinely fiddly and only worth it on latency-critical surfaces. The simpler version of the same idea: if you are running a query rewrite or a classification step, start retrieval on the original query in parallel with the rewrite, and use whichever finishes usefully first.

Measure per stage or none of this applies

Everything above is guesswork without a breakdown. Instrument each stage as a span and record it per request:

const t = tracer.start("retrieval")
const timings: Record<string, number> = {}

async function stage<T>(name: string, fn: () => Promise<T>): Promise<T> {
  const start = performance.now()
  try { return await fn() }
  finally { timings[name] = performance.now() - start }
}

const vector = await stage("embed", () => embed(query))
const hits   = await stage("search", () => vectorSearch(vector))
const ranked = await stage("rerank", () => rerank(query, hits))

await metrics.record({ query_id, timings, total: sum(timings) })

Then look at p50 and p99 per stage, not the mean of the total. The mean hides that one stage is fine most of the time and occasionally takes two seconds — which is usually the embedding API having a bad minute, and which no amount of index tuning will fix.

What not to bother with

Reducing embedding dimensions for speed. Search was 12ms. Halving it saves 6ms and costs retrieval quality.

A faster vector database. Same reason. Migrate for scale, cost or filtering support — not for query latency.

Caching final answers. Tempting and dangerous in multi-tenant systems, where a cache keyed on question text alone will eventually serve one tenant's answer to another. If you cache, key on the question and the full filter set, and see the leakage failure mode.

More in when a reranker is worth the latency, why your RAG returns wrong answers, and retrieval systems that are actually evaluated.