Hybrid search beats a bigger embedding model

The usual response to bad retrieval is to upgrade the embedding model. Larger dimensions, better benchmark scores, a re-index over a weekend.

It generally produces a small, real improvement — and it does nothing at all for the queries that were failing worst, because those queries were never failing for a reason a better embedding model addresses.

Embeddings are built to generalise. The queries that fail hardest are the ones where you needed the opposite: this exact string, not something like it.

The query class that fails

Look at a week of real queries and sort them by whether a human would call the top result correct. In every corpus we have looked at, the bottom of that list is dominated by one shape:

• ERR_4417
• part 88-2210-B
• invoice INV-2026-0831
• Steinhaus (a surname)
• v2.14.3
• section 7.2(b)

These are rare, high-information tokens. An embedding model learned little about ERR_4417 because it appeared almost nowhere in training, so its vector carries almost no specific signal — it lands somewhere in the general neighbourhood of "error code", along with every other error code you have ever documented. The retrieved chunk is topically adjacent and practically worthless.

Meanwhile BM25 — a ranking function from the 1990s — finds it immediately, because it is matching a string against an inverted index and rare terms are exactly what it weights most heavily. The property that makes the token hard for embeddings is the property that makes it easy for lexical search.

They fail on opposite queries

This is why hybrid works, and it is worth stating precisely:

Neither dominates. They are close to complementary, which is unusually good news: fusing them recovers most of both columns.

Fusing them

Do not add the scores. BM25 is unbounded and corpus-dependent, cosine is roughly 0 to 1, and any weighted sum requires you to normalise two incomparable scales and then tune a weight that drifts as the corpus grows.

Use Reciprocal Rank Fusion instead. It throws away the scores and keeps only the ranks, which is what makes it robust:

/**
 * RRF: each list contributes 1/(k + rank) to each document it ranks.
 * k = 60 is the value from the original paper and is a fine default —
 * it damps the influence of the very top ranks so one list cannot
 * unilaterally decide the fused order.
 */
function rrf(lists: string[][], k = 60): string[] {
  const scores = new Map<string, number>()

  for (const list of lists) {
    list.forEach((id, i) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + i + 1))
    })
  }

  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id)
}

const fused = rrf([
  await vectorSearch(query, { limit: 50 }),
  await bm25Search(query,   { limit: 50 }),
])

No normalisation, no weight to tune, no re-tuning when the corpus doubles. A document ranked well by either method surfaces; a document ranked well by both surfaces higher.

You probably already have BM25

This is the part that makes the trade obvious. If your documents are in Postgres, full-text search is already there:

-- One GIN index. No new service, no new dependency.
create index docs_fts on chunks
  using gin (to_tsvector('english', content));

select id, ts_rank(to_tsvector('english', content), q) as score
from chunks, plainto_tsquery('english', $1) q
where to_tsvector('english', content) @@ q
order by score desc
limit 50;

Postgres ts_rank is not textbook BM25 — it does not carry the same length normalisation — but for fusion purposes it is close enough, because RRF only consumes the ordering. If you are already running Postgres for the vectors via pgvector, hybrid search is an index and about thirty lines. Compare that to re-embedding a corpus.

Elasticsearch and OpenSearch give you actual BM25. Most dedicated vector databases now ship a lexical or sparse mode as well.

Then measure it, because the aggregate will disappoint you

Fusion typically moves overall recall@10 by a handful of points, which reads as underwhelming and causes teams to abandon it.

Segment the eval set before drawing that conclusion. Split your questions into conceptual and contains-an-identifier, and score them separately. The aggregate is flat because the conceptual half was already fine. The identifier half is where the movement is, and in the corpora we have measured it is not a few points — it is the difference between mostly failing and mostly working.

It also tends to be the half your users complain about, because a customer quoting an error code from their screen and getting a generic troubleshooting page is a much more visible failure than a slightly off paraphrase answer.

More in why cosine similarity lies, why your RAG returns wrong answers, and retrieval systems that are actually evaluated.