Evaluation is the real product

Every AI project has a moment where it looks finished. The prompt is dialed in, the demo runs clean, the stakeholders nod. The model answered ten questions perfectly in the room. What that moment actually proves is narrow: that the system can succeed once, on inputs someone chose because they already knew it would. It says almost nothing about the ten-thousandth request from a stranger who phrased the question in a way no one anticipated.

The gap between "worked in the demo" and "works in production" is not a model problem. It's a measurement problem. And the thing that closes it — the artifact you build, curate, and defend for the life of the system — is the evaluation set. The model is rented or swapped or fine-tuned away. The eval is what stays.

The one durable asset

Think about what actually changes over the life of an AI feature. The base model gets deprecated and you migrate to a newer one. A prompt that was three paragraphs becomes a structured template. You add retrieval, then rewrite the retrieval. A vendor raises prices and you move to a competitor. Each of those is a swap, and every swap is a bet that the new thing is at least as good as the old thing.

You cannot make that bet on vibes. The only way to know a change is safe is to run both versions against a fixed set of cases with known-good answers and compare. That fixed set is the eval. It is the single component that every other component is measured against, which makes it the one thing you can't afford to treat as an afterthought.

This reframes the whole project. You are not building a model integration that happens to need some tests. You are building an evaluation harness that happens to have a model plugged into it. On the work I care about, the eval set is the first thing I want to agree on, not the last — because it's the thing a team will still be using in a year, long after this quarter's model is a footnote.

Three layers: offline, human, online

A real evaluation practice runs on three layers, each catching what the one before it can't. They are not alternatives; they feed each other.

Offline is the fast layer: a curated set of inputs with graded expected outputs, run automatically on every change. It's cheap, deterministic, and it catches regressions in seconds. Its weakness is that it only knows about cases you thought to include.

Human is the judgment layer: people grading real outputs against a written rubric, blind to which version produced them. It catches the failures offline metrics miss — the answer that scores well on exact-match but reads as curt, evasive, or subtly wrong. Its output isn't just a score; every disagreement it surfaces becomes a new offline case.

Online is the truth layer: what actually happened when real users met the system. Did they accept the answer or edit it? Did the ticket resolve or escalate? Did they come back? These are the only signals that reflect reality instead of a proxy for it — and they're also the noisiest and slowest, which is exactly why you need the other two layers in front of them.

Building the first set from real traffic

The most common way eval sets go wrong is that a well-meaning engineer sits down and invents fifty test cases from imagination. Those cases encode what the author expects users to do, which is precisely the population you don't need to test — you already handle it. The interesting inputs are the ones nobody would think to write down.

So I don't invent the first eval set. I mine it. Real traffic, sampled deliberately: a slice of the easy majority, and then an aggressive over-sampling of everything weird — the long inputs, the empty inputs, the inputs in the wrong language, the ones where the user was clearly frustrated, the ones the current system already got wrong.

# Don't sample uniformly — over-weight the tail, that's where failures live
cases = []
cases += sample(traffic.filter(outcome="resolved"),   n=40)   # the easy majority
cases += sample(traffic.filter(outcome="escalated"),  n=60)   # where it broke
cases += sample(traffic.filter(edited=True),          n=60)   # humans fixed the answer
cases += sample(traffic.filter(latency_p99=True),     n=20)   # slow + strange

for c in cases:
    c.expected = grade_with_rubric(c)   # a human writes the answer of record

Two hundred to three hundred graded cases is usually enough to start — far more valuable than two thousand cases that all look the same. The rubric matters as much as the cases: "correct" is rarely one thing. For a support assistant it might be factual accuracy, plus tone, plus whether it correctly refused when it didn't know. Write those down as separate scored dimensions so a fluent-but-wrong answer can't hide behind a fluent-and-right one.

The ship gate

An eval set that nobody has to pass is just a dashboard. The point is to make it a gate: a change ships only if it holds or improves the score on every dimension that matters, and regressions block the release automatically.

const base = await runEval(mainBranch);
const pr   = await runEval(currentBranch);

// No dimension may regress beyond noise; overall must not drop
const regressed = DIMENSIONS.filter(d => pr[d] < base[d] - TOLERANCE);
if (regressed.length) {
  core.setFailed(`Blocked: regressions in ${regressed.join(", ")}`);
}

The tolerance band is doing quiet, important work. LLM outputs aren't perfectly deterministic, so a naive "score must not decrease" gate fails constantly on noise and teams learn to ignore it. Measure the run-to-run variance first, set the band above it, and now a failure means something — which is the only condition under which people will respect it.

Regression and drift

Two failure modes only the eval catches, and both are invisible day to day. The first is silent regression: you rewrite a prompt to fix one class of error and quietly break another. Without a gate, that trade is undetectable until a user reports it weeks later. With one, it's a red check on the pull request that opened it.

The second is drift. The model didn't change, your code didn't change, but the world did — new products, new slang, new kinds of question the system was never tuned for. The offline set slowly stops resembling live traffic. The defense is the feedback arrow from the online layer: every real failure gets triaged, and the ones that represent a genuine gap get a human-graded answer and join the offline set. The eval isn't a fixed exam. It's a living record of every way the system has ever been wrong, so it can never be wrong that way again unnoticed.

Own the eval, own the system

When a team takes over an AI system, the model is the least of what changes hands. The thing that lets them own it — swap models, tune prompts, add features without fear — is the evaluation harness: the graded set, the rubric, the gate wired into CI, and the loop that keeps the set honest as the world moves. Give a team that, and they can change everything else safely. Withhold it, and every change is a gamble, no matter how good the model underneath.

That's why I treat evaluation as the product and the model as an implementation detail. If you've got an AI feature that works in a demo and you're nervous about touching it, the eval set is almost always what's missing. That's a good conversation to have.