The task mix decides the review bill

Vespper published a DOCX MCP server on 1 September 2026 — it was back on the front page on 28 September — and the product is not the interesting part. The benchmark is. It is documented in more detail than most published model cards, and once you stop reading it as a score and start reading it as a workload, it prices your own pipeline whether the vendor meant it to or not.

The architecture first, because it explains the numbers. Rather than hand the agent OOXML, the agent edits an HTML abstraction and a fine-tuned reconciler maps that intent back onto Word XML. The model receives a triplet — (html_old, html_new, xml_old) — and returns xml_new for one block, and the post is explicit that this is "No reasoning, no tool calls, no loop." Locating the block is a separate deterministic locator, tested on its own, and tracked changes are computed by diffing the reconciled block against the original rather than generated by the model.

That split is the right one. The model does the part only a model can do — decide what the new text should be — and everything with a correct answer is handled by code that can be tested. The reconciler is small by design: "the 3–8B range", LoRA at rank 16, alpha 32, LR 2e-4, AdamW 8-bit, roughly 16k mined training triples, validation pass rate "stable around 0.9". You are not paying a frontier model to shuffle XML.

The vendor's per-task price is a quarter to a third of what a document actually costs you. The rest is review, and how much review is decided by the mix of task types in your own document stream — not by the model you picked.

The table is the useful artefact

The evaluation is better than it needs to be. 279 DOCX editing tasks, six candidates — Vespper's own MCP, SuperDoc MCP v0.18.1, Office CLI v1.0.145, Adeu MCP v3.0.2, Anthropic's DOCX skill, and a plain python-docx harness — each run on two models at medium reasoning. Two methodology decisions worth crediting: the same system prompt for every candidate with no per-solution tuning ("including ours"), and setup steps removed so "Every candidate starts its first turn already able to read and edit the file." python-docx is described as "the floor", which is the right control.

Scoring is block alignment by Longest Common Subsequence, then comparison of content and style. The dataset is 2,046 tasks built from docxcorp.us documents (government, healthcare, finance, legal), split 70/15/15 at the document level to avoid leakage, stratified on topic, with 300 tasks manually labelled and an LLM judge plus a fixer agent used to flag noisy ones.

And then the table that made me open a scratch file. Pass rate by task group, Vespper MCP on GPT 5.6 Sol:

The page labels these rows "groups overlap — one task can touch tables, lists, and paragraphs", which is why the n values sum to 415 against a suite of 279. So this is not a partition and you cannot average it. That is not a criticism — overlapping groups are the honest shape for editing tasks — but it does mean the table has to be read as eight separate statements about eight separate capabilities, not as one distribution.

Two workloads, one product, two prices

Here is why that matters. A per-task pass rate is a statement about a capability. Your per-document cost is a statement about a mix of capabilities, and you get to choose almost none of it — your customers' documents choose it.

Take two plausible streams of 1,000 edit tasks each, and apply the published rates. First, generated report and proposal work:

paragraphs   700 × (1 − 0.755) = 700 × 0.245 = 171.5 failures
tables       200 × (1 − 0.778) = 200 × 0.222 =  44.4
lists        100 × (1 − 0.719) = 100 × 0.281 =  28.1
                                       total = 244.0  →  24.4% need review

Now contract redlining, same product, same model, same system prompt:

paragraphs        400 × 0.245 =  98.0
tracked changes   300 × 0.423 = 126.9
headings          200 × 0.393 =  78.6
hyperlinks        100 × 0.750 =  75.0
                        total = 378.5  →  37.9% need review

Same tool. The review rate moves by 1.55×, and the only variable is which capabilities the work touches.

Now price it. Review at $25/hour with a 60-second review is $0.4167 an item — the slow end of the review-queue arithmetic I have written about before. The model call needs the per-task number the vendor does not publish, but one of their four ratios can be inverted to get it, and only one: Office CLI is quoted at "~$0.24/task and ~57s/task", "~4.5x more expensive and ~3.8x slower than its Vespper counterpart" — same model, so the ratio is tool-versus-tool.

$0.24 / 4.5 = $0.0533 per task      (band over the rounding: $0.052–$0.055)
57 s  / 3.8 = 15.0 s per task

Per 1,000 tasks:

model        1,000 × $0.0533 =  $53.30     $53.30
Mix A review   244.0 × $0.4167 = $101.67  → $154.97 per 1,000 = $0.155/document
Mix B review   378.5 × $0.4167 = $157.71  → $211.01 per 1,000 = $0.211/document

Read the shares. The model is 34.4% of Mix A and 25.3% of Mix B. Review is 65.6% and 74.7%. The number the vendor advertises is the smaller half of the number you have to quote a customer, and that is true of every document-editing tool, not this one.

Read the difference. Mix B minus Mix A is $56.04 per 1,000 tasks against a model bill of $53.30 — the review load created purely by task mix is 5% larger than the entire model spend. At 100,000 documents a month that is $5,604 a month that no architecture decision removes, because it is a property of the paperwork.

The four ratios are not comparable

The other three published ratios each move two things at once, which is the trap in reading a benchmark summary. "~7x cheaper and ~3.7x faster" compares GPT 5.6 Terra with Vespper against GPT 5.6 Sol with the DOCX skill — model and tool both change. "2.7-2.9x cheaper" against the DOCX skill is a tool comparison, and inverting it puts the DOCX skill at $0.144–$0.155 a task. That is real money: at 100,000 tasks the swap to Vespper saves $9,100–$10,100 a month, and it is the larger of the two levers I have priced here.

But be precise about what kind of lever it is. The tool swap is a decision you make once, on a spreadsheet, before launch, and it is the decision the vendor is selling you. The task mix is a bill you pay monthly, decided by whoever sends you documents, and no vendor's headline can tell you what it will be. Both are worth $5,000–$10,000 a month at that volume. Only one of them appears in a launch post.

Match the eval's unit to your consumer's unit

One sentence in the methodology decides whether this benchmark can be reused as your acceptance test:

Style is the computed look after rendering (not Word's formatting XML).

The unit of comparison is the rendered reading experience, with tracked changes applied to content. For "did the document end up saying what the user wanted", that is exactly the right unit, and it is a deliberate choice rather than an oversight.

It is the wrong unit if your .docx is feedstock rather than output. If a template engine reads content controls, an assembly step reads bookmark names, an accessibility checker reads alt text and style names, or your own approval UI diffs the XML, then an edit that renders correctly and renames a bookmark passes this benchmark and breaks your pipeline. Nothing about the vendor's numbers is wrong; the unit simply does not match the consumer. That is the same class of error as reporting one accuracy number across a twelve-field schema — a summary is only meaningful in the unit of the thing that has to be right.

The published limitations list makes the boundary concrete: no creating or replying to comments, no attaching images or videos, and no support for latent styles — "styles that are not defined in styles.xml". If your review workflow lives in Word comments, this is not a tuning problem. It is a stop.

The three rows at the bottom are the three smallest samples

Hyperlinks at 25.0% is the number that would kill this tool in an evaluation meeting. It rests on eight tasks, two of which passed. The 95% Wilson interval around 2/8 is roughly 7% to 59% — which spans "unusable" and "about average". Header/Footer and Images are 50.0% on four tasks each, and those intervals run from about 15% to 85%. Those three rows account for 16 of the 415 task slots in the table.

The two rows the workload is actually made of are Tables (n=90, 68%–85%) and Paragraphs (n=159, 68%–82%). The interval width falls from ~70 points at n=4 to ~13 at n=159, which is just the square root of the sample size doing its job — I have worked through what that means for a pass threshold elsewhere. The practical rule is shorter: before you reject a capability because of a low row, check the n. A 26-point gap on n=8 is a hypothesis. Go and label forty hyperlink edits yourself; they are cheap to find and you will know by Thursday.

What I would actually do with this

1. Get your own task mix before you get a vendor. Sample 200 real edits from your own queue and classify them into these eight groups. That histogram, not the vendor's average, is your review rate.
2. Price the review, not the call. At one task per document the review is 66–75% of the bill. Any comparison that prices only tokens is comparing the small half.
3. Write down the unit of correctness your downstream consumer reads, and check the benchmark compares in that unit.

The genuinely good news in this post is one line I did not expect in a launch announcement: no per-solution prompt tuning, "including ours". A benchmark where the vendor submits to the same harness as everyone else, publishes its worst numbers with their sample sizes attached, and loses two rows outright is reporting rather than marketing. I would rather price a tool from a table like that — including the parts that are unflattering — than from a headline ratio.

This is the work I do on document extraction pipelines that have to survive real paperwork, and it is close to what an AI reliability audit covers when the pipeline already exists and the review bill is the surprise. More on the economics in the review queue is the product, 97% accurate is not a number, and quoting a per-document price you can defend.

The benchmark figures, task-group table, scoring method, dataset description, training hyperparameters and limitations are quoted from Vespper's launch post for its DOCX MCP, published 1 September 2026 and read 28 September 2026. The per-task price of $0.0533, the two workload mixes, the review cost of $0.4167 an item and the per-document totals are my arithmetic on those published rates, not figures Vespper states — the post gives no dollar amount for its own product. The 1,000-task mixes, the one-way-per-document assumption and the 60-second review are a stated scenario.