Changing the embedding model without downtime

A better embedding model ships. Its benchmark scores are higher, it costs the same, and switching looks like changing a string in a config file.

It is not. Vectors from two models occupy different spaces, and the distance between them means nothing. You cannot mix them in one index, you cannot migrate incrementally within one index, and a partial migration produces a system where retrieval quality depends on which model happened to embed each chunk.

There is no gradual embedding migration. There is an old index, a new index, and a cutover — and the only question is what you learn before you make it.

Two indexes, both live

The shape is a standard blue-green deployment applied to a vector store:

// During migration, every write goes to both. Reads go to whichever
// index is currently authoritative.
async function upsert(chunk: Chunk) {
  await Promise.all([
    oldIndex.upsert(chunk.id, await embedV1(chunk.text), chunk.metadata),
    newIndex.upsert(chunk.id, await embedV2(chunk.text), chunk.metadata),
  ])
}

async function search(query: string, filters: Filter) {
  const index = config.activeIndex === "v2" ? newIndex : oldIndex
  const embed = config.activeIndex === "v2" ? embedV2 : embedV1
  return index.query({ vector: await embed(query), filter: filters })
}

Two properties matter here.

The query embedder is bound to the index. The most common migration bug is a query embedded with the new model searching the old index. It returns results — plausible-looking, ranked, entirely meaningless. Nothing errors. Keep the pairing in one place so they cannot diverge.

Chunk ids must be stable across models. If your chunker is deterministic, the same document produces the same ids under both, so backfill is "recompute vectors for known ids" rather than "rebuild the corpus". If chunk ids are ordinal or random, you are re-chunking as well as re-embedding, and now two variables are changing at once.

Backfill as a batch

Re-embedding a corpus has no latency requirement, which makes it the clearest batch API candidate there is — typically half price, at providers that offer it.

Track progress explicitly, because this runs for hours or days and will be interrupted:

alter table chunks add column v2_embedded_at timestamptz;
create index on chunks (v2_embedded_at) where v2_embedded_at is null;

-- Resumable by construction. Restarting picks up where it stopped.
select id, text from chunks where v2_embedded_at is null limit 1000;

The partial index makes the "what is left" query fast even when the corpus is large and nearly complete, which is exactly when you will be running it repeatedly.

Shadow-evaluate before cutting over

This is the step most migrations skip, and it is the reason to build two indexes rather than just rebuilding one.

With both indexes live, run your eval set against both and compare — segmented, not aggregated:

                    v1      v2
recall@10          0.81    0.84
  paraphrase       0.88    0.93   <- the new model is better here
  identifier       0.34    0.29   <- and worse here
  cross-section    0.52    0.58
  table            0.61    0.62

That is a realistic result and it is not a clean win. A newer model can be better overall and worse on a specific query class, and if identifier queries are what your users actually type, the aggregate improvement is misleading. The remedy might be to lean harder on lexical search after cutting over — but you can only know that in advance if you measured the segments.

Then shadow real traffic. Run production queries against both indexes, serve the old results, log both:

const primary = await oldIndex.query({ vector: await embedV1(q), filter })
// Fire and forget. Never on the critical path, never served.
void newIndex.query({ vector: await embedV2(q), filter })
  .then((shadow) => log.shadowResult({ q, primary: ids(primary), shadow: ids(shadow) }))

Real queries surface what a fifty-case eval set cannot: the phrasings you never thought of, the long tail, and the queries where the two indexes disagree completely. Agreement rate on the top result is a useful single number to watch, and the disagreements are the reading list.

Cut over behind a flag, gradually

// Percentage rollout, sticky per user so nobody sees quality flip
// between two consecutive questions in the same session.
const useV2 = hashToUnit(userId) < config.v2RolloutFraction

Sticky assignment matters more than it looks. A user whose first question is answered well from v2 and whose follow-up is answered badly from v1 experiences the product as unreliable, which is worse than either index alone.

Roll to 5%, then 25%, then 50%, watching whatever quality signal you have — thumbs, escalation rate, follow-up-question rate. Keep the old index for at least a week after 100%, because the rollback is only cheap while it still exists.

What it costs to keep both

Two indexes is two lots of storage and two embedding calls on every write during the migration window.

For a corpus of eight million chunks at 1,536 dimensions that is roughly 49 GB per index — so call it 100 GB plus index overhead for the duration. The embedding cost is genuinely small; the storage and the operational attention are the real expense.

Which is the argument for a bounded window. Decide the migration period up front, set a date to delete the old index, and make dual-write a temporary state with an owner rather than a configuration that quietly persists for a year.

When not to migrate

A better model is not a reason on its own. The reasons that hold up:

• The current model is deprecated or being retired.
• A measured, segmented improvement on your eval set, on the query classes your users actually send.
• A meaningful cost or dimension reduction that survives an evaluation.

Not: a higher score on a public benchmark. Benchmark corpora are not your corpus, and the segmented comparison above is the only evidence that transfers.

More in embedding a million documents, a retrieval eval you can build in a day, and retrieval systems that are actually evaluated.