Reranking is 86% of the latency and 15% of the gain

I had been filing "run retrieval on the device" under cost. The embedding call and the rerank call go to zero, so the marginal cost of a query goes to zero, and that is the end of the story as far as the invoice is concerned.

Xtriever is a Rust hybrid retrieval engine that does exactly that. BM25 over tantivy, all-MiniLM-L6-v2 in 384 dimensions stored as int8, reciprocal rank fusion at Σ 1/(60 + rank), then a ms-marco-MiniLM-L-6-v2 cross-encoder over the head of the fused list, with the final order blending the fused and cross-encoder scores half and half. It runs "on the device" and "searches its own index offline, with no server and no network".

What made me stop and open a scratch file is that the README publishes both halves of the trade in one document — a BEIR quality ablation and a latency record from a phone — and the two halves point in opposite directions.

The ablation, with the deltas

nDCG@10 on three BEIR test sets, from the repository's own evaluation harness with the default models:

The table is more interesting in differences than in levels. Taking the mean column:

adding the dense stage to BM25     0.444 → 0.480   +0.036
or against BM25 alone              0.420 → 0.480   +0.060   (+14.3%)
adding the cross-encoder on top    0.480 → 0.491   +0.011   ( +2.3%)

Adding a second retrieval signal to BM25 is worth 5.45× what adding the cross-encoder to the fused list is worth. Or as a share of the ground the pipeline covers over BM25 alone — 0.420 → 0.491, a gain of 0.071 — the fusion step is 84.5% of it and the rerank step is 15.5%.

Per corpus, the rerank step buys +0.007 on SciFact, +0.008 on NFCorpus and +0.020 on FiQA. The fusion step buys +0.029, +0.031 and +0.120. FiQA is where both stages do the most work, and fusion does six times what reranking does there.

What that costs in seconds

The iOS record is on an iPhone 16e, Release, "over 20 measurement queries, re-ranking 10 candidates": a 585 MB index over 427,947 passages "opened in place from the app bundle", 51.5 MB of models, peak phys_footprint of 335 MB "under the project's 600 MB ceiling", a fused list at a median of 0.20 s, a re-ranked list at a median of "1.21 s more", and 0.74 s to open the index.

Price the two stages against each other:

per re-ranked candidate   1.21 s / 10        = 0.121 s
fused list                                 = 0.20 s
re-rank                                    = 1.21 s
                                             --------
total                                      = 1.41 s
rerank share              1.21 / 1.41      = 85.8%

But the recorded run re-ranks 10 candidates and the pipeline default is "20 candidates by default". The README states the scaling rule outright: "Most of a search is the cross-encoder: each re-ranked candidate is one forward pass, whatever the corpus size." So the default, on the same hardware and the same stated linearity:

20 candidates             20 × 0.121       = 2.42 s
total                     0.20 + 2.42      = 2.62 s
rerank share              2.42 / 2.62      = 92.4%

15.5% of the quality, 85.8% of the latency at the recorded candidate count, 92.4% at the documented default. And 2.62 s is 1.86× the recorded 1.41 s for twice the candidates.

A cross-encoder is the only stage in a retrieval pipeline that reads the query and the passage together, which is why it recovers negation and qualifiers. It is also, on this evidence, where nearly all of the query time goes — and it is not close.

The result that inverts the usual intuition

The README also runs the same queries over a 20,000-passage Wikipedia slice. It "opens faster on the same phone (0.45 s) and fuses faster (0.16 s), but re-ranks no faster."

427,947 passages down to 20,000 is a 95.3% smaller corpus. Index open falls 39.2%, from 0.74 s to 0.45 s. The fused list falls 20%, from 0.20 s to 0.16 s. The rerank stage does not move at all.

Everyone I know who tunes retrieval latency tunes the index — smaller chunks, fewer vectors, a coarser quantisation, an ANN parameter that trades recall for speed. This measurement says the index is not where the query time goes. The cross-encoder's cost is candidate count times one forward pass, and corpus size does not appear in that expression. The only knob on the dominant stage is how many candidates you hand it, which is a number in your code rather than a property of your data.

Recall is the number the ablation hides

One more line, and it is the one I would put at the top of the file if it were mine: "Recall@100 of the default pipeline: 0.955, 0.321 and 0.706."

Mean 0.661. The high value is 0.955, the low is 0.321 — a spread of 2.98× across three corpora, on the same engine, with the same models, measured by the same harness that produced the ablation above it. On the low corpus, 67.9% of relevant passages are not in the top 100 at all, which means no candidate count and no reranker can ever reach them. The reranker reorders what retrieval returned. It is a precision stage, and that is a recall number.

This is what I would carry into my own pipeline. A mean nDCG@10 of 0.491 and a recall@100 of 0.955 on the corpus you happened to test describes that corpus. The same code on another corpus of the same shape can be structurally unable to answer, and every number in the ablation — including the rerank delta — will look fine while it happens. I have argued this from the metric side before; this is the first published table I have seen where the same pipeline's recall varies by 3× across the test sets printed directly underneath it.

The degradation path is a published row

The README is explicit about what happens under pressure. "It degrades instead of failing." If the dense stage or the re-ranker errors, "or runs out of its time budget", the search "returns the previous stage's results and says so in a stage report." A strict mode returns the error instead.

Which means the fallback's quality is not a mystery. Serving the previous stage's results means serving the fused row: 0.480 against 0.491. The degraded mode costs 1.21 s less and 2.3% of mean nDCG@10, and both numbers are already in the table.

The failure mode worth thinking about is that a time budget is exceeded when the device is busy — thermally throttled, another app holding the CPU — so the queries that degrade are the queries served under load. Quality and load move inversely, on the surface where the person waiting is least patient, and the only signal is a stage report. If you do not surface that report, you have a system whose accuracy varies with conditions you cannot see, and you will discover it the way I discover most things, which is from a user who noticed.

What the free retrieval is actually worth

On-device removes the retrieval fee outright, and that is real. I put a hosted reranker at roughly $1–2 per 1,000 queries in when a reranker is worth the latency, so 100,000 queries a month is $100–200 that this design does not spend.

It is also not the bill. Price the half the device cannot do — the generation call — at Claude Sonnet 5 rates checked today, $2/MTok input and $10/MTok output, over a stated shape of five 900-token chunks plus 800 tokens of instructions in and 400 tokens out:

input      5,300 × $2  / 1,000,000  =  $0.01060
output       400 × $10 / 1,000,000  =  $0.00400
                                       --------
per query                              $0.01460
100,000 queries                        $1,460

$1,460 against $100–200. The retrieval fee is 7–14% of the generation cost on the same volume, so making retrieval free is worth doing and does not change the size of the bill. And on a corpus with NFCorpus-shaped recall you pay that full $0.01460 to generate from a context that is missing two-thirds of the material. The expensive failure is not the reranker. It is a confident answer produced from a candidate set that never contained the answer, billed at full price — which is the argument in why RAG returns wrong answers arriving from the other direction.

What I would change, and what I am not taking from this

Three things, none of them exotic. Set the candidate count from your own eval rather than accepting the default, because it is the only input the dominant stage has. Record recall@100 per corpus next to nDCG, because the ablation cannot tell you that one of your corpora is unreachable. And log the stage report, because a silent fallback to the fused list is a quality regression that raises no error.

What I am not taking from this is a benchmark result I can cite. The project is "0.1.0, not published" — nothing is on crates.io, PyPI, Swift Package Index or Maven yet. The planned learned ranker "is not built" and its crate is a placeholder. Android has "only emulator records so far". The wasm32 build "fails today, a known and tracked gap". And the recall sentence does not name its datasets; the three figures appear in the same order as the quality table's columns, so I have read them as SciFact, NFCorpus and FiQA. If the order is different, the named corpus changes and the 2.98× spread does not — which is the part I am actually using.

The latency record is 20 measurement queries, which is a thin sample for a median, and I would treat 1.21 s as an order of magnitude rather than a measurement of your phone — the same caution as how many runs before a number means anything. The per-candidate figure is the durable part, because the README states the linearity rather than leaving me to infer it.

What I keep coming back to is the shape of the trade rather than the engine. A retrieval pipeline is a stack of stages that each buy some quality for some latency, and the last one on the stack is usually the one with a model in it and a name you recognise. Here it buys 15.5% of the gain and pays 86% of the time, and the stage underneath it — the unglamorous fusion of two cheap rankers — buys five times more for a fraction of that. That ordering is worth checking in your own pipeline before you tune anything, and it takes an afternoon with a retrieval eval to see it.

This is the kind of stage-by-stage measurement I would rather run deliberately than discover from a user complaint, and it is squarely what a retrieval system that is evaluated rather than vibe-checked looks like from the inside. The same arithmetic on the generation side is in what an AI feature costs to build and when the model bill triples.

All Xtriever figures and quotations — the five-row nDCG@10 ablation and its harness, "Recall@100 of the default pipeline: 0.955, 0.321 and 0.706.", the iPhone 16e Release record over "20 measurement queries, re-ranking 10 candidates" (585 MB index, 427,947 passages, 51.5 MB of models, 335 MB peak phys_footprint, 0.20 s fused, "1.21 s more" re-ranked, 0.74 s index open), the "20 candidates by default", "each re-ranked candidate is one forward pass, whatever the corpus size.", the 20,000-passage slice that "re-ranks no faster", the degradation contract, and the stated limitations — are from github.com/mirth/xtriever, read 27 September 2026. Model rates are from claude.com/pricing, checked the same day. The 20-candidate extrapolation is mine, on the README's own stated one-forward-pass-per-candidate linearity. The 5-chunk, 900-token context shape and the 100,000-queries-a-month volume are my stated scenario, not measurements — substitute your own token counts and the ratios hold while the dollars move.