Masking columns is not masking data

dbmask went up on Hacker News on 10 September 2026. It is a Python CLI, MIT licensed, that does three things to a SQL database: scan it for sensitive columns, mask them with deterministic fakes, then validate that the masking worked. The validate step is the interesting part and the reason I bothered installing it — almost nothing in this category tries to prove it did its job. It is on PyPI as dbmask 0.1.1, it needs Python 3.9 or newer, and it talks to Postgres, MySQL, SQL Server, Oracle and SQLite through SQLAlchemy.

Everyone building on an LLM eventually needs this. You want a copy of production to test an extraction pipeline against, or to hand a contractor, or to paste into a prompt while you debug. So you mask it. Then the masked copy becomes the eval fixture, and the fixture becomes the thing your accuracy numbers are computed from, and nobody looks at it again.

I built a support-tickets table of the shape every AI product actually has — structured contact columns plus a free-text body where the customer typed their own name — and ran the full workflow against it.

The masked database passed every check the tool has, including --strict, while every customer's real name, one customer's real email address and one customer's real phone number were still sitting in it in plain text.

The scan

Five rows. Columns customer_name, email, phone, body, invoice_total, received_on. Default config, no overrides:

[UNKNOWN ?] main.tickets.id (inconclusive, conf=0.00)
[UNKNOWN ?] main.tickets.customer_name (inconclusive, conf=0.00)
[SENSITIVE] main.tickets.email -> email (pattern, conf=1.00)
[SENSITIVE] main.tickets.phone -> phone (pattern, conf=0.80)
[UNKNOWN ?] main.tickets.body (inconclusive, conf=0.00)
[UNKNOWN ?] main.tickets.invoice_total (inconclusive, conf=0.00)
[SENSITIVE] main.tickets.received_on -> phone (pattern, conf=0.80)

Three things in seven lines.

customer_name is inconclusive. Not a bug — detection samples values and matches patterns, and human names have no pattern. An email address has an @ and a TLD. A name is just text. This is the correct behaviour of a value-based detector and it is also the reason no value-based detector will ever find the names in your database.

body is inconclusive, which is the same fact restated. The README says so plainly: detection is column-level, and mixed PII in free text is out of scope. That limitation is honestly documented and it is still the one that will hurt you, because free text is where an AI product keeps its data.

received_on — a date column, values like 2026-04-03 — matched the phone rule at 0.80 confidence. It is about to be masked.

Apply, then validate

after : {'email': 'kimberly.rodriguez581@example.invalid', 'phone': '+675 09 497 6280', 'received_on': '2488-92-04'}
after : {'email': 'stephanie.baker895@example.invalid',   'phone': '+87 83 0758 7863',  'received_on': '1218-08-20'}
after : {'email': 'michael.lewis63@example.invalid',      'phone': '+52 771 7224 7878', 'received_on': '4511-49-05'}

2488-92-04. Month 92. The format_random strategy preserves the shape ####-##-## and fills it with digits, which is exactly right for a phone number and produces a calendar impossibility for a date. Rows 4 and 5 came out 4185-83-92 and 7775-91-52.

Then the check that is supposed to catch all of this:

[✓] row_count              main.tickets: Row counts match (5).
[✓] schema_elements        main.tickets: columns/types match.
[✓] masking_completeness   main.tickets.email: PK-aligned comparison of 5 row(s): every sensitive value differs from its source.
[✓] masking_completeness   main.tickets.phone: ...
[✓] masking_completeness   main.tickets.received_on: ...

--- Validation summary ---
  pass    : 10

RESULT: PASSED ✓

Ten passes, exit 0. --strict, which exists so that "could not verify" fails the gate, also returns PASSED ✓. And here is what is in the database that passed:

(1, 'Aoife Ní Bhraonáin', 'kimberly.rodriguez581@example.invalid', '2488-92-04')
(2, "Jean-Luc O'Farrell-Mbeki", 'stephanie.baker895@example.invalid', '1218-08-20')
(3, '李伟', 'michael.lewis63@example.invalid', '4511-49-05')

1 Hi, this is Aoife Ní Bhraonáin. My invoice 8841 shows 1.234,56 EUR but I paid 1.234,56 on 03/04/2026. Call me on +353 87 555 0134.
2 Jean-Luc here (jl.ofarrell@example.co.uk). Charged twice, GBP 89.00 each, ref 8842.

Every name intact. Row 2's original email address still in the body, one column over from the fake one that replaced it. Row 1's original phone number, likewise. The email column was masked; the email address was not.

This is not a tool failure. The tool did what it says on the tin and warned me four times that four columns were unclassified. It is a workflow failure, and it is the default one, because a green checkmark ends an investigation. That is what green checkmarks are for.

Marking the free text sensitive makes it worse

The documented remedy is a field-overrides file. So I forced customer_name and body to sensitive with the name rule, and told it received_on is safe:

(1, 'Rbshn Dr Qxrniyxgw', 'Ce, lgvd jj Hlcie Qq Qvjzmmxgq. Qr ntxskxk 2141 pofzl 3.179,51 KOX fhc V dzlv 7.014,06 pf 75/39/8962. ...')
(3, '李伟',                '我的发票 0316 金额 ¥92,217.43 有误。联系人 李伟。')

Row 3's name survives a column explicitly marked sensitive, in both places. The character substitution has no mapping for CJK, so it rewrites the digits and leaves every ideograph untouched — 李伟 is byte-identical to the source. Any masker that works by substituting characters from a Latin alphabet will do this; it is the same class of bug as an ASCII-only slug function, except the output is a compliance incident rather than a bad URL.

And validate's verdict on that row:

[✗] masking_completeness   main.tickets.customer_name: 1 row(s) hold a value identical to the source — these values bypassed masking.
[✓] masking_completeness   main.tickets.body: PK-aligned comparison of 5 row(s): every sensitive value differs from its source.

It caught the name column, because there the whole value was unchanged. It passed the body column, because the digits inside it changed — so masked != source is satisfied while 李伟 is still readable. That is the real lesson, and it generalises past this tool: completeness checks test that a value changed, which is a strictly weaker property than the sensitive thing being gone. For a full-column replacement those are the same property. For free text they are not, and nothing in the output tells you which case you are in.

What it does to your eval set

Take the leak out of it for a moment and look at what came back.

received_on is now 2488-92-04. If that column is ground truth for a date field, your accuracy on dates is meaningless, and your date parser — the one handling 03/04/2026 ambiguity — will throw on a month that cannot exist, which is a stack trace that exists only in your test environment. That is a day of debugging a bug you do not have.

Five emails became first.last###@example.invalid. Five samples of one template, ASCII only, one domain. aoife@nibhraonain.ie and li.wei@example.cn are gone and so is everything they were testing.

And in the overrides run, 1.234,56 EUR became 3.179,51 KOX and 03/04/2026 became 75/39/8962. The document text your extractor reads is now noise, so a golden set built on it is scoring your model against characters drawn at random. Masking that is safe enough to satisfy a reviewer is usually destructive enough to invalidate the measurement, and the two jobs are almost never the same job.

What I would do

1. Treat every UNKNOWN as sensitive until a human says otherwise. Run dbmask scan --json, fail the pipeline on a non-empty unknown list, and make the overrides file the artefact that records the decision. The tool already refuses to mask what it cannot classify; the gap is that nothing refuses to ship it.
2. Never let a column-level masker near free text. Drop the column from the fixture, or replace whole values with synthetic text you wrote. A partially scrambled sentence is the worst of both — still identifying, no longer readable.
3. Write your own leak assertion. One query, and it is the only check that tests the property you care about: take every distinct name, email and phone from the source, and assert none of them appears anywhere in the masked copy, across every text column. It found all five of mine in seconds.
4. Separate the safe copy from the eval fixture. The safe copy exists so nobody sees a customer. The fixture exists so a number means something. Build the fixture from synthetic documents you control, and keep the real distribution — the unicode names, the 1.234,56, the ambiguous dates — because those cases are the entire value of a real sample and masking is precisely the operation that removes them.
5. Check the false positives, not just the misses. received_on -> phone at 0.80 confidence is a column being silently destroyed by a tool you ran to protect it. Read the scan output. It is seven lines.

Everything above is from dbmask 0.1.1 installed from PyPI and run locally on CPython 3.9.6 / SQLAlchemy 2.0.52 / SQLite 3.51.0 on 11 September 2026, against a five-row SQLite database I seeded; the licence, supported engines and the column-level limitation are from the project's own README, checked the same day.

If a copy of production is feeding your evals, that is the first thing I pull apart in an AI reliability audit — a pipeline measured against a corrupted fixture is a pipeline with no measurement at all, and it is the most common reason an extraction system that scores well goes wrong in production. Related: field-level accuracy metrics and why demos lie.