Embedding a million documents

Embeddings are cheap enough that people stop estimating them, which is fine until the corpus is a million documents, someone changes the chunking strategy, and the whole thing has to be rebuilt on a deadline.

The token cost is usually the smaller half. The part that surprises people is storage, and storage is decided by a parameter chosen early and rarely revisited.

Embedding a corpus is cheap. Embedding it four times because nobody planned for re-indexing is the actual cost, and so is storing vectors at a dimension nobody chose deliberately.

The token arithmetic

A million documents averaging 2,000 tokens is 2 billion tokens. At OpenAI's published rates:

text-embedding-3-small  ($0.02/MTok):  2,000 × $0.02  =   $40
text-embedding-3-large  ($0.13/MTok):  2,000 × $0.13  =  $260

Forty dollars to embed a million documents. That is the number that stops people estimating, and for a one-off it is correct — this is genuinely cheap.

Halve it again with the batch API where the provider offers one; embedding a corpus has no latency requirement at all, which makes it the clearest possible batch candidate.

Then count how many times you will do it

The recurring cost is not the initial index. It is everything that forces a rebuild:

Three of those four are full rebuilds. A team that embeds a corpus four times in the first eighteen months has spent $160 rather than $40 — still trivially cheap, and the point is not the money.

The point is the time. Two billion tokens through a rate-limited API is a job measured in hours or days, not minutes, and it usually needs to happen while the old index keeps serving. That is a dual-write migration, and it is worth knowing you will need one before you are doing it under pressure.

Storage is the number people miss

A vector is a list of floats. At 32 bits each:

1536 dimensions × 4 bytes = 6,144 bytes per vector

Chunk that million documents into 8 chunks each and you have 8 million vectors:

8,000,000 × 6,144 bytes ≈ 49 GB

Forty-nine gigabytes of vectors, from $40 of embedding. Plus the index structure — HNSW typically adds 30–100% on top — plus the original text, plus metadata. Call it 100 GB, and managed vector databases are priced on stored volume and memory rather than on how cheap the embedding was.

The larger models are worse: 3,072 dimensions doubles all of the above.

Three levers on storage

Dimension reduction. Some embedding models support truncating the vector without retraining — 3,072 down to 1,024, or 1,536 down to 512. Quality falls, usually less than expected, and storage falls proportionally. Measure it against your eval set rather than accepting a benchmark:

dims    storage    recall@10
1536      49 GB        0.81
 768      25 GB        0.80
 384      12 GB        0.76

That shape is common — a large saving for almost nothing down to about half, then a cliff. Where the cliff sits is corpus-specific, which is why this is a measurement rather than a rule.

Quantisation. Storing 8-bit integers instead of 32-bit floats is a 4× reduction with a small quality cost. Binary quantisation is 32× with a large one, usable as a first-stage filter with exact rescoring on the survivors.

Fewer chunks. Halving the chunk count halves the storage. Not a free lever — it trades against retrieval precision — but worth noticing that a chunking decision made for quality reasons is also a storage decision.

Choose a model you can leave

The migration cost is what makes model choice consequential, more than the price per token.

Vectors from different models are not comparable. You cannot mix old and new embeddings in one index, so switching means re-embedding everything and running two indexes during the transition:

// Dual-write during migration. Both indexes live; queries hit the old
// one until the new one is complete and evaluated.
await Promise.all([
  oldIndex.upsert(id, await embedV1(text)),
  newIndex.upsert(id, await embedV2(text)),
])

Which argues for a few things at setup time: record the model and its version on every stored vector, keep the source text so re-embedding never requires re-fetching from the origin, and keep chunking deterministic so the same document produces the same chunk ids under a new model.

That last one is what turns a migration from "rebuild everything" into "recompute vectors for known ids", which is a considerably calmer operation.

Where it stops being cheap

Two cases where the arithmetic changes:

Embedding queries, not documents. At a million queries a month, each ~20 tokens, that is 20 MTok — pennies. But it is on the critical path, and the constraint is latency rather than cost.

Very large corpora. A hundred million documents is 200 billion tokens, which is $4,000 on the small model and a serious storage bill. At that scale self-hosting an open-weight embedding model on your own GPUs becomes cheaper than the API, and the trade becomes an infrastructure question rather than a line item.

Below roughly ten million documents, embedding cost is not worth optimising. Storage and migration planning are.

More in the document changed, the index did not, what an AI feature costs to build, and retrieval systems that are actually evaluated. OpenAI embedding rates from their published pricing, checked 27 August 2026.