Why we sell a fixed-scope audit first

Someone has an AI feature in production that is behaving badly. They want it fixed. The natural next step is a proposal for a build engagement — several weeks, five figures, a scope, a start date.

Both parties are being asked to commit to something neither can price. The client does not know whether this firm is any good. The firm does not know what is actually wrong, because nobody has measured it.

A fixed-scope audit is not a sales tactic. It is the only way to price the real engagement, and it is cheap enough that being wrong about it costs nobody much.

What is unknown at the point of quoting

An AI feature that is "confidently wrong sometimes" has at least six plausible causes, and they have completely different costs to fix:

• The test set never resembled production — selection, not sampling
• Retrieval never returns the right passage, so the prompt cannot help
• Retrieval works and the model answers from prior knowledge anyway
• There is no schema, so malformed output flows downstream silently
• There is no confidence signal, so nothing can be routed to a person
• The whole thing is fine and the accuracy target was never achievable on those inputs

The first is a week. The last is a conversation about scope rather than a build. Quoting before knowing which one you have means either padding heavily — the client overpays — or guessing low and absorbing the difference.

What the audit produces

A bounded engagement with a fixed price and a defined deliverable. Not open-ended discovery, and not a document of recommendations.

A measured baseline. A hundred stratified cases, labelled, with per-field or per-query accuracy. Frequently the first real number anyone has had.

The diagnosis. Which of the causes above it is, with evidence. For retrieval that means recall@k and an ablation separating retrieval failure from generation failure. For extraction it means the per-field table showing which two fields carry the errors.

A cost model. What it costs to run now, and what the fixes do to that — including retries, which are usually invisible and frequently double digits.

A ranked list of fixes, each with effort and expected movement. Ranked by return, not by interest.

The eval harness itself, in the client's repository, running in their CI. This is the part that outlives the engagement — it is theirs whether or not they work with us again, and it is the thing that makes any future change verifiable. See the eval set is the deliverable.

Why fixed scope rather than a retainer

A retainer is the natural shape for ongoing work and the wrong shape for this one, for a reason that cuts both ways.

For the client: a retainer with no defined output is an open-ended spend on a supplier they have not worked with. A fixed price with a fixed deliverable is a bounded bet, and the deliverable has value even if they never engage further.

For us: a fixed scope forces the diagnosis to be time-boxed. An hourly arrangement rewards thoroughness past the point of usefulness, and the honest version of this work is a small number of days.

The retainer is the right shape after — once there is a system with an eval suite and a known operating profile, ongoing work is genuinely ongoing. Starting there inverts the order.

The outcome we do not hide from

Some audits end with a recommendation not to build.

The system is fine and the expectation was wrong. Or the accuracy target is not reachable on those inputs at any budget, and the real answer is a narrower scope or a different process upstream. Or the corpus is small enough that the whole retrieval architecture is unnecessary and a cached prompt does the job.

That is a successful audit. The client paid a small fixed amount and avoided a five-figure build that would not have worked, and they have a measurement proving it rather than an opinion.

It also means the audit cannot be a formality that always concludes with a proposal. If it did, it would be worth nothing as evidence, and the whole mechanism — small commitment, real information — collapses.

What it is not

Not a free consultation. Producing a labelled sample and a measured baseline is real work. Free discovery calls produce the version based on a description rather than the data, which is the input that generates the estimate everyone later regrets.

Not a code review. Reading the code tells you what it does, not how well it does it. The measurement is the point.

Not a pilot. No new feature is built. The output is knowledge and a harness, not a prototype — a pilot that "shows it can work" reproduces exactly the selection problem that caused the original trouble.

Who it is not for

Honest limits, because a bounded engagement mis-sold is worse than none:

• Nothing in production yet. Nothing to measure. That is scoping, which is a different engagement.
• The problem is already diagnosed and agreed. Then skip to the build; paying to confirm a known answer is a waste.
• No access to representative data. Without real documents or real queries, there is no sample, and without a sample there is no measurement — only opinion, which is available free.

More in seven questions for an AI contractor, hiring an AI engineer vs bringing in a firm, and the AI reliability audit.