How to break an AI demo in ten minutes

A demo is a curated sample presented by someone who chose it. That is not dishonest — it is what a demo is. The problem is that for AI systems the gap between a curated sample and production is much wider than for ordinary software, and the demo format hides exactly the thing you need to evaluate.

Four requests turn a demo into evidence. None is hostile, and all four are reasonable enough that a vendor who has shipped this will have anticipated them.

Ordinary software either works or throws. An AI system fails by producing something plausible, and a demo is the format least able to show you that.

1. "Run it on this."

Bring your own documents. Not your cleanest ones — a realistic sample, including the photograph someone took at an angle and the supplier whose template is strange.

Do it live, in the meeting.

Good answer: they run it, some fields come back flagged or uncertain, and they talk you through why. Uncertainty surfacing on hard documents is the system working.

Concerning: "we'd need to fine-tune on your data first." Sometimes true. Frequently it means the demo is tuned to the demo documents, and the honest version of that sentence is "it does not generalise yet."

Disqualifying: the demo cannot accept arbitrary input at all.

2. "Show me one it gets wrong."

Ask them to find a failure case, live.

Good answer: they produce one quickly, because they know where the edges are. "Handwriting in the amount field, here — we route these to review rather than guessing."

Concerning: a long pause and then a trivial failure. Either they have not looked, or they are choosing the least alarming one.

Disqualifying: "it doesn't really get things wrong." Every extraction system gets things wrong. A vendor who will not say where means you will discover it from your own customers.

This is the highest-value question of the four. A team that has operated a system has a ready list of its failure modes; a team that has demoed one does not.

3. "What did it do to be uncertain about that?"

Pick any output and ask how confident the system is and why.

Good answer: a per-field confidence, an explanation of what it is derived from, and a threshold that decides routing. Ideally the source — this page, this region.

Concerning: a single document-level score. One number for a whole document is not actionable, and a vendor presenting it as their uncertainty story has not thought about the operating model.

Disqualifying: no confidence at all, or a confidence number nobody has calibrated. Ask directly: in the 90-95% confidence band, how often is it actually right? If there is no answer, the number is decoration.

4. "What happens to the ones it can't do?"

Ask to see the exception path. Not a description — the actual interface a reviewer uses.

Good answer: they show you a queue. There is a cropped image of the uncertain field, a proposed value, and a keyboard shortcut. They know how many seconds a review takes, because that number determines the operating cost.

Concerning: "they'd show up in a report." A report is not a workflow, and nobody reads it after week two.

Disqualifying: no exception path. This means low-confidence values are being written into your system indistinguishable from good ones — which is the failure this whole category of software is supposed to prevent.

Two things that look like red flags and are not

A range instead of a price. A vendor who says "somewhere between X and Y until we see a hundred of your documents" is being accurate. The one who quotes a precise figure from three samples is guessing with more confidence.

An accuracy number lower than a competitor's. Accuracy claims are only comparable if the test sets are. A vendor quoting 94% on a stratified sample including photographs is likely better than one quoting 99% on clean native PDFs. Ask what the number was measured on — the answer separates the two immediately.

The pattern

All four questions ask the same thing in different clothes: have you operated this, or only built it?

Building something that extracts invoices well on twenty invoices is a weekend now. Knowing what it costs at forty thousand a month, what it does with the photograph, how fast a reviewer clears the queue, and which band of confidence is actually trustworthy — that only comes from having run one.

If the answers are specific and some are unflattering, that is the signal. A demo where everything works and every answer is reassuring is a demo, and you will find the rest out later.

More in seven questions for an AI contractor, when extraction accuracy collapses, and the AI reliability audit.