The eval set is the deliverable

Contracts for AI work carefully assign ownership of the code. The code is the least valuable thing being produced.

Prompts get rewritten within months. Models are replaced within a year, and the pipeline around them is a few thousand lines of fairly ordinary engineering. What persists is the labelled data: the documents somebody read carefully, the correct answers somebody wrote down, and the questions somebody decided were worth asking.

The eval set is the only artefact of an AI engagement that cannot be rebuilt from a specification. Whoever holds it decides whether you can change anything safely.

Why it is the expensive part

A hundred labelled documents is a day or two of careful human work, and the value is not the hours. It is the judgement encoded in them: which documents are representative, which edge cases matter, what "correct" means for a field where two readings are defensible.

That judgement came out of conversations with the people who handle these documents today. It is domain knowledge, written down in a form a machine can check, and it is the reason a new engineer can change a prompt without breaking something quietly.

Without it, every change is a guess. With it, every change is a measurement. That is the entire difference between maintainable and not.

The lock-in shape

A vendor who keeps the eval set does not have to say anything about lock-in for it to exist. It arrives naturally:

• You cannot verify their accuracy claims independently.
• You cannot evaluate a competitor on equal terms, because you have no test set to run.
• Your own team cannot safely change the prompt after handover.
• Rebuilding the set costs what it cost the first time, and by then the people who understood the edge cases have moved on.

Nobody has to intend this. A vendor with the eval set in their own repository, running in their own CI, has produced it as a side effect of normal working — and the effect is the same as if they had planned it.

What to specify

Plain language, in the contract, before work starts:

Deliverables include, and the Client owns outright:

• the labelled evaluation dataset, including source documents, per-field correct values, and the sampling rationale;
• the evaluation harness that runs it, and the CI configuration that gates deploys on it;
• all prompt templates and their version history;
• documentation of the accuracy achieved per field, and the method used to measure it.

These are delivered in the Client's repository, in open formats, and are updated on the same cadence as the code. No component of the evaluation pipeline may depend on infrastructure the Client does not control.

The last sentence is the one that does the work. An eval harness that runs on the vendor's platform is not a deliverable you own, whatever the ownership clause says.

Three things people forget

The source documents, not just the labels. A JSON file of correct answers is useless without the documents they refer to. If the corpus is confidential and lives with the client, this is fine and should be explicit. If the vendor assembled the sample from client data, the sample itself has to come back.

The sampling rationale. Why these hundred documents? A stratified set has a structure — source type, quality bands, supplier distribution — and without the reasoning nobody can refresh it correctly. A set nobody can refresh decays into an archive.

The failure log. Every document that went wrong in production and was added to the set afterwards. This accumulates during the engagement and is the highest value part, because those are real failures rather than anticipated ones.

Read it as a signal

How a vendor responds to this clause is informative before you have signed anything.

Good: they raise it first. Firms that have done this a few times know the handover is where engagements go wrong, and they have a standard answer.

Fine: they agree without much thought. Most contractors have not considered it and have no objection once it is named.

Worth pausing on: hesitation, or an argument that the eval set is "proprietary methodology". The labels are your documents and your domain knowledge. The harness is sixty lines of glue.

Disqualifying: an eval set that only exists inside their platform, sold as a feature.

The reciprocal obligation

Fair to state the other side, because ownership without maintenance is a technicality.

A client who owns an eval set and never updates it has an asset that expires. Documents change, suppliers change templates, the input mix shifts. A set assembled in March describes March, and by the following March it is measuring a world that has moved.

So the handover has to include the practice, not just the files: who refreshes the set, at what cadence, and what triggers adding to it. Ours is a quarterly refresh of a slice plus every production failure added as it occurs — which is cheap, because the labelling was already done by whoever reviewed the failure.

If nobody on the client side will own that, the honest thing is to say so during scoping rather than to hand over an artefact that will be stale before it is first needed.

More in seven questions for an AI contractor, white-label AI work, and how to evaluate an LLM pipeline.