The cost model nobody builds until it's too late

There's a familiar moment about three months into a successful AI launch. The feature works, usage is climbing, everyone's pleased — and then the monthly bill arrives and someone in finance asks a question nobody can answer: what does this thing cost us per use? Silence. The number was never measured, because in the demo it was rounding error, and by the time it wasn't, the architecture that determines it had already set like concrete.

This is the cost model nobody builds until it's too late. Not because it's hard — it's arithmetic — but because it feels premature during the exciting part and unfixable by the time it's urgent. The teams that avoid the trap treat cost the way they treat quality: as a number they measure from day one, on every request, and design against. Here's what that actually looks like.

Cost per what, exactly?

The first discipline is picking the right unit. "How much do we spend on AI a month" is an accounting question and it's useless for engineering, because it moves with volume and hides everything interesting. The unit that matters is cost per meaningful outcome — per resolved ticket, per generated draft, per completed try-on — because that's the number you compare against the value of that same outcome.

For a language-model feature, the base of that number is token accounting, and it's lopsided in a way people forget: output tokens usually cost several times more than input tokens. A request that reads a large context and writes a short answer is cheap; one that writes a long answer is not, even from the same model.

// Prices are illustrative $/1M tokens — output dominates
const IN = 3.00, OUT = 15.00;

function requestCost(inTok: number, outTok: number) {
  return (inTok / 1e6) * IN + (outTok / 1e6) * OUT;
}

requestCost(2400, 380);   // ≈ $0.0129 — just the primary call

That figure is where most estimates begin and end. It's also wrong — not in the math, but in the scope. It counts one clean call and ignores everything a real request drags along with it.

The costs nobody counts

A production request is rarely one model call. It's a pipeline, and every stage has a price tag that the napkin estimate omits. Retries multiply the primary call by their retry budget on the unlucky fraction of requests. A RAG feature pays for an embedding of the query, then a reranker pass over dozens of candidates, before the generation call even starts. Guardrail or moderation passes are extra calls. And the whole thing sits on infrastructure — vector store, cache, queue — with its own monthly floor.

Then there's the cost that lives entirely off the request path and surprises people most: evaluation. If you're doing the job right, you run an eval suite on every change, and a large graded set run through a capable model on every pull request has a real monthly cost of its own. It's money well spent — it's what lets you change anything safely — but it belongs in the model, not as a shock later. The point isn't that any one of these is large. It's that they're invisible individually and decisive in aggregate.

The tail is where the money goes

Averages lie about AI costs, because the distribution of request sizes is heavy-tailed. Most requests are small and cheap; a few are enormous — the user who pastes a whole document, the conversation that runs forty turns deep, the retrieval that pulls maximum context. Those tail requests can cost fifty times the median, and if you budget on the mean you'll be badly wrong the moment usage skews toward the tail.

This is why per-request logging beats monthly totals. When you record the cost of every single request, you can see the distribution — median, p95, p99 — and the tail stops being a nasty surprise and becomes a design input. It also tells you exactly where to point a cap: a hard ceiling on input size, a max-turns limit on a conversation, a token budget per user per day. You can only place those limits intelligently if you can see the shape of what you're limiting.

// Attach the real cost to every request — this is the whole trick
await costLog.record({
  requestId, feature,
  inTok, outTok,
  rerankerCalls, retries,
  usd: fullyLoadedCost({ inTok, outTok, rerankerCalls, retries }),
});
// now p50 / p95 / p99 per feature are a query, not a guess

Cost as a first-class metric

Here's the shift that changes everything: cost is not a finance concern that arrives at month-end. It's an engineering metric that lives next to quality, latency, and error rate, measured on the same requests, watched on the same dashboards. When a pull request changes the prompt, the eval gate should report not just what happened to quality but what happened to cost per request — because the two trade against each other constantly, and a change is only good if you can see both sides of the trade.

This is the same discipline I bring to evaluation generally: a thing you don't measure is a thing you can't manage, and a thing measured only in aggregate is a thing you'll manage too late. Cost per request deserves a gate as much as quality does — a change that quietly triples the token count should be as visible as one that drops accuracy.

The levers, in order

When the number is too high, there's a rough order of operations, cheapest and safest first. Start by not making the call at all: cache aggressively on inputs that fully determine the output, and a large share of traffic can become free hits — the effect I lean on hard in Voilae, where a repeated request returns instantly for nothing. Next, make each call smaller: tighter prompts, fewer and better-ranked retrieved chunks, shorter outputs where the task allows.

Then route by difficulty. Most requests don't need the most expensive model; a smaller, cheaper model handles the easy majority, and only the hard cases escalate — often more than halving cost with no visible quality loss, provided your eval set proves the smaller model actually holds up. Only after those do you negotiate rates or commit to volume discounts. The mistake is reaching for the last lever first, buying a bulk discount on a workload you never optimized.

A cost model is a design tool

The reason to build this early isn't bookkeeping — it's that the cost model changes what you build. When you know, in week one, that a feature has to land under two cents a request to clear its value, that constraint shapes the architecture: it tells you whether you can afford the big model or need to route, whether you must cache or can skip it, how much context you can afford to retrieve. Discover the same constraint in month three and every one of those decisions is already made, expensively, in the wrong direction.

A cost estimate built alongside the first prototype is one of the highest-leverage documents in the whole project. It's a page of arithmetic that routinely reveals a feature is a non-starter before a quarter is sunk into it, or — just as usefully — that there's ten times more headroom than anyone assumed and you can afford to be more ambitious. Either way, you learned it while the design was still soft enough to act on.

Build it in the first week

None of this requires special tools. It requires deciding that cost per outcome is a number you own from the start: estimated before you build, logged on every request once you do, gated in CI alongside quality, and treated as a design constraint rather than a monthly surprise. The teams that do this are never ambushed by the invoice, because they've been watching the number climb in real time and shaping the system around it the whole way.

Building that model — the honest, fully-loaded, per-outcome kind — is one of the first things I reach for on any AI feature, and it's often the artifact that keeps paying off long after launch. If you're shipping something and can't yet say what it costs per use, that's the number worth pinning down first.