Chunk size is not your problem

When a RAG system answers badly, chunk size is the first thing almost everyone tunes. It is a single number, it is easy to change, and there is a great deal written about it.

Run the sweep properly — 256, 512, 1024, 2048 tokens, measured against a real eval set — and the result is nearly always the same: a few points of movement across the whole range, with no dramatic optimum. Meanwhile the queries that were failing badly are still failing badly, because they were failing for reasons chunk size does not touch.

Chunk size is a smooth trade-off with a broad optimum. The failures people attribute to it are usually cliffs somewhere else.

What the number actually trades

There is a genuine tension, and it is worth understanding before dismissing it:

Smaller chunks produce a more precise embedding, because the vector represents less text and each phrase contributes more of its direction. They also strip context — a chunk saying "this does not apply to enterprise customers" is useless without knowing what this is.

Larger chunks carry their own context, so a retrieved passage is more likely to be self-contained. But one vector now averages several distinct ideas, and the specific clause you needed contributes proportionally less. Retrieval gets fuzzier, and you pay more tokens per retrieved chunk.

That is a real curve. It is also a shallow one. Between 400 and 1,200 tokens most corpora behave similarly, which is why the sweep disappoints.

The three cliffs people blame on chunk size

Structure split across a boundary. A table's header row lands in chunk 12 and its data rows in chunk 13. Chunk 13 is now a grid of numbers with no column meanings — unretrievable by any query that names a column, and misleading if retrieved. Same for a numbered list split mid-sequence, or a code block cut in half.

This is not a size problem, it is a boundary placement problem. Chunking on structure — headings, sections, table boundaries — rather than on a token count fixes it at any size:

// Split on document structure first, then on size only within a section,
// and never inside a table or code block.
function chunk(doc: ParsedDoc, target = 800): Chunk[] {
  return doc.sections.flatMap((section) => {
    const atomic = section.blocks.filter((b) => b.kind === "table" || b.kind === "code")
    // Atomic blocks stay whole even when they exceed the target.
    // An oversized table is better than two useless halves.
    return packBlocks(section.blocks, target, { keepWhole: atomic })
  })
}

Rare identifiers. ERR_4417 fails to retrieve at 256 tokens and at 2,048 tokens, because the problem is that embeddings generalise away exactly the tokens that are rare. No chunk size fixes it; lexical search alongside does.

Answers that span sections. "How does the refund policy differ between plans?" requires the consumer terms and the enterprise addendum together. There is no chunk size at which one chunk contains both — they are in different documents. The fix is retrieving more chunks and reranking, or a query decomposition step, not a bigger window.

Add context instead of adding size

The strongest fix for the small-chunk context problem is not larger chunks. It is keeping the chunk small for the embedding and giving it its context back separately.

Prepend the breadcrumb. Embed the chunk with its document title and heading path in front of it. Costs a few tokens, and it means a chunk reading "This does not apply to enterprise customers" embeds as Refund Policy › Exceptions › This does not apply to enterprise customers, which is a completely different and far more useful vector.

Retrieve small, return large. Index at 300 tokens for retrieval precision, then expand what you send the model to the surrounding section:

const hits = await search(query, { limit: 20 })   // small, precise chunks
const context = await Promise.all(
  hits.slice(0, 5).map((h) => expandToSection(h.chunkId)),
)
// Matched on a precise vector; answered from a passage with its context intact.

This decouples the two jobs that a single chunk size was being asked to do at once, which is why it beats any point on the original curve.

Overlap, modestly. 10–15% overlap between adjacent chunks catches sentences that straddle a boundary. Beyond about 20% you are paying storage and retrieval cost for duplicated text and crowding your top-k with near-identical results.

If you are going to sweep, sweep properly

Chunk size is worth measuring once, when you set up the corpus. Do it against the eval set and read the segmented output, not the aggregate:

                    256    512   1024   2048
recall@10          0.71   0.78   0.79   0.74
  paraphrase       0.82   0.88   0.91   0.89
  identifier       0.31   0.33   0.30   0.28   <- flat. not a size problem.
  cross-section    0.44   0.49   0.52   0.55   <- rises. more context helps.
  table            0.38   0.41   0.39   0.36   <- flat, and low. boundaries.

That shape is typical, and it is far more informative than the top row. The aggregate says "1024 is marginally best". The breakdown says identifier queries need lexical search, table queries need structural chunking, and cross-section queries need reranking or more retrieved chunks — three different fixes, none of which is a number in a config file.

Pick something between 500 and 1,000, chunk on structure, add breadcrumbs, and go and work on whichever row of that table is lowest.

More in why cosine similarity lies, why your RAG returns wrong answers, and retrieval systems that are actually evaluated.