Why cosine similarity lies to you

A support bot is asked: what is the refund window for enterprise customers?

Retrieval returns three chunks. The top hit, at 0.89, is from the consumer terms: "Refunds are available within 30 days of purchase." The correct chunk, from the enterprise addendum — "Enterprise agreements supersede the standard refund window; refunds are available within 90 days" — comes back third at 0.81.

The model answers 30 days. Confidently, with a citation, from a real document in your corpus.

Nothing failed. The retriever did exactly what it is built to do, and what it is built to do is not what you needed.

What the number actually measures

Cosine similarity is the angle between two vectors. Those vectors are produced by an embedding model that was trained to place text with similar distributional meaning near each other. That is a real and useful property, and it is not the same as "answers this question".

In the example above, the consumer chunk is dense with the query's own vocabulary — refund, window, days, purchase. The enterprise chunk spends half its length on contractual framing. On pure topical resemblance the wrong chunk genuinely is closer. The score is correct; the objective is wrong.

Three failure classes this produces

Negation is nearly invisible. "Enterprise customers are eligible for a refund" and "Enterprise customers are not eligible for a refund" differ by one token and describe opposite worlds. In embedding space they are neighbours, because they are about the same thing. Any corpus with exceptions, exclusions or eligibility rules — which is to say any policy corpus — is exposed to this.

Qualifiers get averaged away. A chunk's embedding is one vector for the whole chunk. The longer the chunk, the more each specific clause is diluted by everything around it. "Supersedes the standard window" is the single most important phrase in the enterprise chunk and it contributes a small fraction of that chunk's direction.

Rare exact strings are the weakest case. Part numbers, error codes, version strings, surnames. Embeddings are built to generalise, and a token that appeared rarely in training has little learned structure to generalise from. Searching for ERR_4417 frequently returns chunks about other error codes, which are topically adjacent and completely useless.

The scores are not comparable across queries

This is the part that quietly breaks production systems. A team notices bad answers, adds a relevance floor — discard anything below 0.75 — and ships it.

Cosine scores are not calibrated. Their absolute value depends on the embedding model, the query's length and phrasing, and how the corpus happens to be distributed. A well-phrased question against a tight corpus might see its best true hit at 0.62; a vague question against a broad one might see a useless chunk at 0.86. A fixed threshold discards good results for some queries and admits junk for others, and because it does both at once the aggregate metrics barely move.

If you want a threshold, derive it per query from the distribution of that query's own results — a gap between the top hit and the rest is informative in a way that the top hit's absolute value is not:

// A relative gate. Keep chunks close to the best hit for THIS query,
// rather than chunks above some global constant.
function gate(hits: Hit[], ratio = 0.9): Hit[] {
  if (hits.length === 0) return hits
  const best = hits[0].score
  return hits.filter((h) => h.score >= best * ratio)
}

That is still a heuristic. It is a heuristic whose failure mode is comprehensible, which the global constant is not.

What to do instead

Cosine similarity is a good first-stage filter and a bad final answer. The fix is structural, not a better embedding model:

1. Filter on metadata before you search. If the user is an enterprise customer, restrict the candidate set to enterprise documents. This turns the worked example above into a non-problem, and it is a where clause.
2. Run lexical search alongside it. BM25 finds ERR_4417 because it is matching the string, not a learned representation of it. Fuse the two result sets — see hybrid search beats a bigger embedding model.
3. Rerank the top 50 down to the top 5. A cross-encoder reads query and chunk together instead of comparing two independently-computed vectors, which is precisely what recovers negation and qualifiers.
4. Measure retrieval separately from generation. If you cannot say what your recall@10 is, you cannot tell whether an answer was wrong because retrieval missed or because the model ignored what it was given. Those have opposite fixes.

Step four first, honestly. Every other change is unverifiable without it, and building the harness takes about a day — here is the smallest version that works.

More in why your RAG returns wrong answers and retrieval systems that are actually evaluated.