Seven questions for an AI contractor

This is written by an AI contractor, which you should factor in. The questions below are the ones that would be most inconvenient for a firm that has demoed this work rather than shipped it — including, in a couple of places, for us.

The general shape: ask about failure, cost and handover. Anyone can talk about the happy path.

Every one of these has an answer that sounds good and an answer that is specific. The specific one is usually less flattering, which is how you tell them apart.

1. How do you measure whether it is working?

Listen for: a golden set, per-field or per-query metrics, a number gating deploys in CI.

Worry about: "we test it thoroughly", "the model is very accurate", a demo offered as the answer.

A team without an eval harness cannot tell you whether their last change helped. They are shipping on impressions, and so will you be. Ask to see the harness — it is a file, and it either exists or it does not.

Our answer: we build one in the first week and it gates the pipeline. It starts small — around fifty labelled cases — because the eval that exists beats the one you designed.

2. What does it cost to run at my volume, and show the arithmetic?

Listen for: tokens per unit of work, published per-token rates, output counted separately from input, retries included.

Worry about: a monthly figure with no working, or "the API costs are negligible".

Cost per document or per conversation should be computable before a line is written. If they cannot produce it in the meeting, they have not done it, and you will discover the real number in month three.

Where our answer is unflattering: the estimate has a range, and the range is wide until we see your actual documents. Anyone quoting a precise figure before seeing your data is guessing with more confidence than we are.

3. What happens when the model gets it wrong?

Listen for: confidence thresholds, a review queue, validation rules, fail-open versus fail-closed as a deliberate choice.

Worry about: "we use a really good model", "accuracy is 97%".

97% of what, measured how, on which documents? And more importantly — what happens to the 3%? A system with no answer to that has decided your customers will find the errors.

Our answer: deterministic validation first — arithmetic and cross-field checks catch more numeric errors than any confidence score. Then per-field routing to a review queue. The queue is a deliverable, not an afterthought.

4. Can you tell me why it produced a specific output?

Listen for: transcripts, retained inputs, prompt versioning, span-level citations.

Worry about: logs that record events without inputs.

This question arrives eventually from compliance, from an enterprise customer, or from a lawyer, and retrofitting it is expensive. The trace has to be designed in.

5. What did you get wrong on your last project?

Listen for: a specific technical mistake, what it cost, what changed.

Worry about: "we're perfectionists", a story where the client was the problem, or genuine inability to name one.

Anyone who has shipped this has been wrong in an interesting way. The answer is also a working sample of how they will talk to you when something breaks.

Our answer: we scored prospects by multiplying rule points by the model's confidence, which made scores incomparable between prospects and quietly sorted by how legible a website was. The rewrite made confidence a gate rather than a multiplier.

6. Who owns the evals, the prompts and the data?

Listen for: you do, in writing, including the labelled test set.

Worry about: vagueness, or a platform that holds the eval set.

The labelled data is the expensive artefact. It took human hours, it encodes your domain knowledge, and it is what lets the next engineer — theirs or yours — change anything safely. A contract where the vendor keeps it is a contract that makes leaving expensive by design.

Get it explicit: prompts, eval sets, labelled data, and the harness are yours, in the repository, on delivery.

7. What would you tell me not to build?

Listen for: a real recommendation against something, with a reason.

Worry about: enthusiasm for everything you mentioned.

Someone who has done this a few times has opinions about which AI features are not worth building. A contractor who agrees your entire wish list is a good idea is selling hours.

Our answer, frequently: you may not need retrieval. If the corpus fits in a context window, a cached prompt beats a vector database on cost, accuracy and every operational axis. That conversation shrinks the project, which is the point of asking.

The pattern

Every question above is really asking the same thing: have you operated this, or only built it?

Building an AI feature that demos well is a solved problem and getting cheaper every month. Operating one — knowing what it costs, what it does when it fails, how to prove what it did, and how to change it without breaking it — is the part that is still rare, and it is the entire difference between the two kinds of firm you are choosing between.

If the answers are specific and some of them are unflattering, that is a good sign. If they are all reassuring, ask question five again.

More in hiring an AI engineer vs bringing in a firm, what an AI feature costs to build, and fractional AI engineering.