The same invoice, uploaded three times

A customer uploads a folder of invoices. The upload stalls, so they upload it again. Then a colleague emails the same invoices to your ingestion address.

Three copies of every document. Each one costs a full extraction, and each one lands in the ledger as a separate payable — which is worse than the wasted money, because now somebody has to work out which of the three duplicates is real.

Deduplication is not a nice-to-have in a document pipeline. Without it, the same invoice can be paid twice, and that is a category of bug customers do not forgive.

Filenames are not identity

invoice.pdf is the most common filename in the world. Two suppliers both send invoice.pdf and the naive dedupe silently drops one — a false positive that is far worse than a duplicate, because a missing invoice is invisible.

The reverse also fails: the same document arrives as invoice.pdf, invoice (1).pdf, and Scan_2026_08_14.pdf.

Filenames carry no identity. Neither does upload timestamp, nor size, nor the Message-ID of the email that carried it.

Level one: the byte hash

import { createHash } from "node:crypto"

const bytesHash = createHash("sha256").update(fileBuffer).digest("hex")

Exact-duplicate detection, free, and it catches the re-upload case completely — the same file uploaded twice produces identical bytes.

Store it with a unique constraint so the database does the arbitration:

create table documents (
  id          uuid primary key,
  bytes_hash  text not null,
  tenant_id   uuid not null,
  filename    text,
  received_at timestamptz not null default now(),
  -- Scoped per tenant. Two customers legitimately having the same
  -- document is not a duplicate; it is two documents.
  unique (tenant_id, bytes_hash)
);

The tenant scoping matters. A shared supplier sends the same PDF to two of your customers, and a global unique constraint would drop the second one.

Level two: the content hash

Byte hashing misses the case that actually costs money. The same invoice:

• Re-scanned, so every pixel differs
• Re-exported from the accounting system, with a new PDF creation timestamp in the metadata
• Photographed instead of scanned
• Passed through a compression step by an email gateway

Byte-identical: no. Same invoice: yes. Paying it twice: also yes.

So hash the content after extraction, over the fields that identify the document rather than over its rendering:

/**
 * Identity of an invoice = who issued it and which invoice it is.
 * Deliberately excludes anything that varies between renderings:
 * scan quality, file size, our own confidence scores, timestamps.
 */
function contentHash(inv: Invoice): string {
  const identity = [
    normaliseSupplier(inv.supplier),   // casing, legal suffixes, whitespace
    inv.invoiceNumber.trim().toUpperCase(),
    inv.issuedOn.toISOString().slice(0, 10),
    inv.total.amount.toString(),        // minor units, integer
    inv.total.currency,
  ].join("�")

  return createHash("sha256").update(identity).digest("hex")
}

Normalisation is where this succeeds or fails. ACME Ltd, Acme Limited and ACME LTD. are one supplier, and if your normaliser does not know that, the content hash differs and the duplicate survives. Keep the normaliser small, tested, and versioned — because changing it invalidates every hash you have stored.

The two-level flow

async function ingest(file: Upload, tenantId: string) {
  const bytes = createHash("sha256").update(file.buffer).digest("hex")

  // Level 1: exact re-upload. Free, no extraction at all.
  const existing = await db.maybeOne(
    `select id from documents where tenant_id = $1 and bytes_hash = $2`,
    [tenantId, bytes],
  )
  if (existing) return { status: "duplicate", of: existing.id, cost: 0 }

  // Only now spend money.
  const extracted = await extract(file)
  const content = contentHash(extracted)

  const sameContent = await db.maybeOne(
    `select id from documents where tenant_id = $1 and content_hash = $2`,
    [tenantId, content],
  )
  if (sameContent) {
    // Extraction already paid for. Record the relationship rather than
    // discarding — a second copy is evidence, not noise.
    await db.query(
      `insert into document_variants (canonical_id, bytes_hash, filename)
       values ($1, $2, $3)`,
      [sameContent.id, bytes, file.name],
    )
    return { status: "duplicate_content", of: sameContent.id }
  }

  return await store(extracted, { bytes, content, tenantId })
}

Level one is free and catches most duplicates. Level two costs one extraction and catches the rest. Recording variants rather than deleting them means you can answer "why is this invoice not in the system?" with "it is, under a different filename, here."

Near-duplicates are not duplicates

The case that deserves care: an invoice and its corrected reissue. Same supplier, same number, different total. The content hash differs — correctly — so both are stored.

That is right, and it is not enough, because now there are two payables for one obligation. Detect the relationship explicitly:

-- Same supplier and invoice number, different content. Almost always
-- a correction, occasionally a supplier reusing numbers. Either way,
-- a human should see it before both get paid.
select supplier_id, invoice_number, count(distinct content_hash) as versions
from documents
group by 1, 2
having count(distinct content_hash) > 1;

Route those to review with both versions side by side. This is one of the highest value checks in an accounts payable pipeline and it costs a nightly query.

Idempotency past ingestion

The same principle applies downstream, and the content hash is what carries it. Keying side effects on the document's content hash means a reprocess cannot double-post:

// Ties into the general pattern — the key is derived from the intent,
// and the intent here includes which document and which prompt version.
const key = `post_to_ledger:${contentHash}:${promptVersion}`
await once(key, () => postToLedger(record))

Including promptVersion is deliberate: a deliberate reprocess under a new prompt should be able to produce a corrected posting, while an accidental reprocess under the same prompt should not. That distinction is exactly what reprocessing after a prompt change turns on, and it falls out of the key for free.

More in a schema that fails loudly, the agent sent the email twice, and how we build extraction pipelines.