Filter first, then search

A support assistant is asked about the refund window and returns the 2023 policy. The 2023 policy is in the corpus, it is genuinely about refunds, and it is a better lexical and semantic match than the current one because the current one is phrased more cautiously.

No retrieval technique fixes this, because retrieval is doing its job. The document should never have been a candidate.

Most retrieval work tries to rank the right answer above the wrong ones. It is cheaper to make the wrong ones ineligible.

Search a smaller haystack

The instinct when results are bad is to improve ranking — better embeddings, reranking, query rewriting. All useful, all working on the same over-broad candidate set.

Filtering attacks it from the other end. If the user is on the enterprise plan, consumer terms are not candidates. If the question is about the current policy, superseded versions are not candidates. This is a where clause, it is exact rather than probabilistic, and it runs before the expensive part:

const hits = await index.query({
  vector,
  topK: 10,
  filter: {
    tenantId: { $eq: user.tenantId },
    status:   { $eq: "current" },        // not superseded
    audience: { $in: ["all", user.plan] },
    lang:     { $eq: user.locale },
  },
})

Inside the query, always. Filtering the results afterwards has completely different behaviour and a much worse failure mode.

The metadata is decided at ingestion

This is why filtering gets skipped: it is not a retrieval-time feature. The fields have to exist on every chunk, which means extracting them when you index — and retrofitting means a full re-index.

The set worth having from the start:

type ChunkMeta = {
  // Access. Non-negotiable, and the reason to get this right on day one.
  tenantId: string
  audience: string[]          // plans, roles, regions

  // Recency and validity. The single most common cause of confidently
  // wrong answers in a corpus that has any history at all.
  status: "current" | "superseded" | "draft"
  effectiveFrom?: PlainDate
  effectiveTo?: PlainDate

  // Provenance and shape.
  docType: "policy" | "contract" | "faq" | "runbook"
  sourceId: string
  version: string
  lang: string

  // Structure, for both filtering and breadcrumbs.
  section: string[]           // heading path
}

status earns its place more than any other field. Corpora accumulate. Old handbooks, previous contract versions, deprecated runbooks — all indexed, all plausible, all wrong. A status filter removes an entire class of failure that no reranker addresses, because the superseded document genuinely is the better semantic match for a question phrased in its own vocabulary.

Deriving filters from the question

Static filters — tenant, plan, language — come from the session. Others depend on what was asked, and a small classification step is enough:

type QueryFilters = { docType?: string[]; status?: string; asOf?: PlainDate }

async function deriveFilters(q: string, session: Session): Promise<Filter> {
  // Cheap model, tight schema, cached prompt. Tens of milliseconds.
  const derived = await classify<QueryFilters>(q, FILTER_SCHEMA)

  return {
    // Session-derived filters are NOT negotiable and are applied last
    // so a classifier can never widen them.
    tenantId: { $eq: session.tenantId },
    audience: { $in: ["all", session.plan] },
    ...(derived.docType ? { docType: { $in: derived.docType } } : {}),
    status: { $eq: derived.status ?? "current" },
  }
}

The ordering in that object is the point. A classifier is a model, and a model can be talked into things by a hostile question. Access filters are applied from session state after the derived ones, so nothing the classifier returns can widen the visible set. Getting a docType wrong costs recall; getting tenantId wrong costs a customer.

Fall back before you return nothing

Filters can over-constrain. A user asks about a topic that only appears in a superseded document, and a strict status: current filter returns nothing.

Empty results are worse than slightly-off results, so widen deliberately and tell the reader:

let hits = await search(vector, strict(filters))

if (hits.length < 3) {
  // Relax the softest constraint first. Never relax access constraints.
  hits = await search(vector, { ...strict(filters), status: undefined })
  usedFallback = true
}

Then surface it: "No current policy covers this. The closest match is from the 2023 handbook." That is a genuinely useful answer, and it is only possible because status was on the chunk in the first place.

Never relax tenantId or audience. The distinction between soft filters, which trade recall, and hard filters, which are access control, should be explicit in the code — a hard and soft split in the filter builder makes it impossible to relax the wrong one by accident.

Measure it as a segment

Add filter-sensitive cases to your eval set and tag them, because the aggregate will not show this:

                       no filters   with filters
recall@10                    0.74           0.81
  superseded-trap            0.22           0.94   <- the whole story
  cross-plan                 0.51           0.89
  general                    0.88           0.87   <- slightly worse, fine

Two things to read there. The trap rows move enormously, which is the case for filtering. And the general row goes down very slightly — occasionally a filter excludes something that would have helped. That is the real trade, it is small, and it is worth making.

More in why cosine similarity lies, the RAG bug that shows tenant A tenant B's data, and retrieval systems that are actually evaluated.