When a reranker is worth the latency

Vector search compares two vectors that were computed separately. The query was embedded without knowing the document; the document was embedded months ago without knowing the query. Whatever the score means, it was produced without either side seeing the other.

A cross-encoder reads them together — query and passage concatenated, in one forward pass, scored for relevance to each other. That is a fundamentally stronger question to be asking, and it is why reranking recovers the failures that no amount of embedding-model upgrading does.

It also costs you 150–400ms and a per-query fee, on every query.

Reranking is the only stage in a retrieval pipeline that gets to read the query and the passage at the same time. Everything before it is an approximation built for speed.

What it recovers

Three things, specifically — and they are the three that cosine similarity is structurally bad at.

Negation. "Enterprise customers are not eligible" and "Enterprise customers are eligible" are neighbours in embedding space and opposites to a cross-encoder, which is reading the sentence rather than comparing summaries of it.

Qualifiers. In a 900-token chunk, the clause "except where superseded by a master agreement" contributes a small fraction of one averaged vector. A cross-encoder attends to it directly when the query is about exceptions.

Ranking within a good candidate set. Vector search often puts the right chunk in the top 20 but at position 11. If you pass the top 5 to the model, that is a miss. Reranking is very good at this specific job — reordering candidates that are all roughly on-topic — because that is the only thing it is asked to do.

That last point sets the whole cost/benefit. Reranking cannot find what retrieval never returned. It is a precision stage, not a recall stage.

The budget, worked

A rerank stage over 50 candidates, per query:

At 100,000 queries a month a hosted reranker is on the order of $100–200 — small next to the generation cost of the same queries, which is the honest comparison. The dominant cost is latency, not money.

Whether 200ms matters depends entirely on the surface:

• Streaming chat. Time-to-first-token is what users perceive, and generation is already 300–800ms. An extra 200ms upstream is roughly 25% on a number nobody is timing. Almost always worth it.
• Type-ahead or instant search. The interaction budget is around 100ms total. Do not rerank; it does not fit.
• Batch or asynchronous. Latency is free. Always rerank.

Retrieve wider when you rerank

The mistake is bolting a reranker onto an unchanged pipeline — retrieve 10, rerank 10. That reorders ten candidates and cannot improve recall at all.

The point of a reranker is that it lets you retrieve wider than you could otherwise afford. Vector search at k=50 is nearly as cheap as k=10; what you could not previously do was hand 50 chunks to a model. Now you do not have to:

// Cast wide, because first-stage recall is what bounds the whole system,
// then let the reranker do the precision work.
const candidates = await hybridSearch(query, { limit: 50 })
const ranked = await rerank(query, candidates)
const context = ranked.slice(0, 5)

Measure it as two numbers. Recall@50 before reranking is your ceiling — the reranker can never beat it. Precision in the final 5 is what the reranker is actually improving. Reporting one blended figure hides which stage is limiting you.

Gate it, and skip it when you can

Not every query needs reranking, and the ones that do are identifiable cheaply. When the first-stage results are already decisive, reordering them is spending 200ms to change nothing:

/**
 * If the top hit dominates its neighbours, the ordering is already
 * confident and a reranker will almost certainly agree. Skip.
 */
function needsRerank(hits: Hit[]): boolean {
  if (hits.length < 3) return false
  const margin = hits[0].score - hits[2].score
  return margin < 0.05          // scores bunched: genuinely ambiguous
}

const ranked = needsRerank(candidates)
  ? await rerank(query, candidates)
  : candidates

On typical traffic this skips 30–50% of queries — the unambiguous ones — while keeping the reranker for exactly the cases it helps. Tune the margin against your eval set rather than accepting the number above; the right value depends on your embedding model's score distribution.

What it will not fix

Be clear about the ceiling before spending the latency:

• Missing content. If the answer is not in the corpus, nothing helps.
• First-stage misses. If recall@50 is 0.6, the reranker's best possible outcome is 0.6. Fix retrieval first — usually with hybrid search — then rerank.
• Generation problems. If the right chunk is in the context and the model answers wrong anyway, that is a prompting problem and reranking is irrelevant.

Which is the order of operations: measure recall, fix the first stage, then add reranking as a precision layer on top of a candidate set that already contains the answer. Adding it to a pipeline with 0.6 recall buys almost nothing and is the most common way this stage disappoints.

More in a retrieval eval you can build in a day, why your RAG returns wrong answers, and retrieval systems that are actually evaluated.