Why evaluation is harder than training.

Every few months a team arrives at the laboratory with the same story. They have built something with a language model and it works — they have seen it work. There is a recording of it working. The deck is persuasive, the demo lands, leadership is enthusiastic. Then someone asks the only question that matters in production: how do you know it still works today? And the room goes quiet.

This essay is an attempt to explain that silence, and to offer a way out of it. The short version: over the last few years, training became industrial while evaluation stayed artisanal. Foundation models moved most of the capability work out of the applied team's hands. What used to be "training" is now prompting, retrieval, and the occasional fine-tune — fast, cheap, and accessible to any competent engineer. The hard part did not disappear. It moved. The scarce discipline in applied AI today is not producing behaviour; it is measuring it.

Why evaluation is structurally harder

It is tempting to treat evaluation as an afterthought — a QA phase bolted onto the end of a build. That instinct comes from classical software, where correctness is mostly deterministic, and from classical ML, where a held-out test set and a single accuracy number carried you surprisingly far. Generative systems break both assumptions, for four reasons.

01The objective is not written down anywhere. Training optimises a proxy — next-token likelihood, preference scores. Your system's real objective is something like "drafts a compliant client letter that a senior associate would sign". That objective lives in the heads of your best practitioners, is fuzzy at the edges, and differs subtly between departments. Before you can measure anything, you have to do the unglamorous anthropology of writing it down.

02The output space is open-ended. A classifier could be graded by lookup. A paragraph cannot. Correctness becomes a judgement — of tone, of faithfulness to sources, of what was omitted — and judgements need judges, and judges need calibrating.

03The ground moves. Model versions change underneath you. Prompts accrete edits. User behaviour adapts to the system, which changes the input distribution you evaluated against. An evaluation is a photograph of a moving object: useful, dated the moment it is taken, and in need of a schedule.

04Someone will ask you to prove it. In regulated industries — where much of our work sits — "it seems fine" is not an answer you can give an auditor. You need evidence that a change did not degrade the system, and a trail showing who decided what, on what basis, when.

A working taxonomy

In practice we build evaluation as five layers, cheapest first. Each layer catches what the previous one cannot see.

01Deterministic checks. Schema validity, citations that actually resolve, numbers that add up, formats, banned content, leaked personal data. These are unit tests for behaviour: fast, free at the margin, and non-negotiable. It is remarkable how many production incidents are caught by checks a first-year engineer could write — and how many teams skip them because they are not "AI evaluation".

02Golden sets. A curated collection of real cases — questions your users actually asked, documents your system actually mishandled — with adjudicated correct answers. Not a public benchmark. Public benchmarks measure the model; golden sets measure your system on your task, which is the only thing your users experience. A good golden set is small (a hundred and fifty well-chosen cases beat five thousand scraped ones), versioned like code, and grows by ritual: every incident, every escalation, every "that's not quite right" from an expert becomes a case. The set is the institutional memory of your failures.

03Model-graded evaluation. Using a language model to judge outputs at scale. Genuinely useful, and the layer most often abused. Judges drift between versions. Judges favour fluent confidence over hedged correctness. Judges can be gamed by style. The rule in the laboratory: an uncalibrated judge is an opinion, not a measurement. Before a judge's scores count for anything, we measure its agreement against human labels on a sample, we track that agreement over time, and we re-calibrate whenever the judge's underlying model changes. Where agreement is weak, the judge is demoted to a triage tool — good for routing cases to humans, not for signing off on them.

04Shadow mode. Run the system against live traffic without letting it act, and compare what it would have done against what actually happened. This is the only honest preview of production, because it is the only layer that samples the world as it is, rather than as your test set remembers it. It is also the layer teams most resist, because it delivers weeks of humbling data at precisely the moment everyone wants to launch. That is the point.

05Human adjudication. Structured, sampled review by people qualified to disagree — with double-rating on a subset so you can measure how much your experts disagree with each other. (They will. That number is your ceiling: no automated metric can be more reliable than the humans it is calibrated against.) Expert attention is the scarcest resource in the whole stack; the architecture of evaluation is largely the craft of spending it where the stakes justify it.

The failure modes

Four patterns account for most of the evaluation theatre we are asked to repair. Benchmark theatre: quoting a public leaderboard as evidence about your task; the correlation is weaker than anyone wants to believe. Goodharting: the moment a metric becomes the target, teams optimise the metric — retrieval scores rise while answers get worse. Rotate metrics, keep some held out, and audit the winners. Judge drift: the silent version bump that moves every score half a point and invalidates every historical comparison; pin versions and re-baseline deliberately. Eval rot: the golden set frozen at launch while the product moved on, measuring last year's system with last year's cases — accurate, precise, and irrelevant.

What regulation teaches

Working under regulators forces a discipline that every team should steal regardless of sector: evaluation precedes capability. Metrics are defined before features are built. Changes ship through gates that compare against the previous baseline. Evaluation sets are versioned artefacts with owners. Decisions are logged with their evidence. None of this slows a team down once it is habitual — it is the teams without it that grind to a halt, usually in the week after their first serious incident.

The order of operations we use on every new system: write down what failure means, with concrete examples, before writing a line of code. Lay the deterministic floor. Build a fifty-case golden set before launch and grow it forever. Shadow the system against reality for weeks, not days. Introduce model judges only after humans have calibrated them. Publish the scorecard internally, weekly, whether it flatters the system or not.

The second principle of this laboratory is: if you can't measure it, you can't ship it. Training — in its modern, diminished sense — produces capability. Evaluation produces reliability. The market has learned to be unimpressed by the first. It pays for the second.