Where AI actually earns its keep

The most expensive AI mistakes I see aren't technical. They're mistakes of selection: a capable team spends two quarters building something impressive that was never going to pay for itself, while the boring problem three desks over — the one that would have returned its cost in a month — sits untouched because it didn't sound like AI.

Whether a model works is mostly a question of execution, and execution is a solved discipline. Whether a problem was worth solving is a question of judgment, and it's where projects actually live or die. Before I write a line of code, I run a candidate problem through three questions. They're simple, they're not about the technology, and they filter out most of what people bring to the table.

Is the feedback loop tight?

The single best predictor of an AI project's success is how quickly you find out you were wrong. A problem where you can see the outcome in seconds — the suggested reply was accepted or edited, the classification was confirmed or overridden, the draft shipped or got rewritten — is a problem you can improve. Every interaction is a graded example, and the system gets measurably better because you can measure it.

Contrast that with a problem where the outcome only becomes visible months later, filtered through so many other factors you can't attribute it — "did the AI-generated strategy memo improve the business?" You can't tune what you can't measure, and you can't measure what takes a year to resolve. This is why the durable early wins cluster around the same verbs: draft, triage, classify, extract, summarize. Not because they're glamorous, but because you learn whether they worked before lunch.

What does being wrong cost?

Models are wrong sometimes. That's not a defect to be engineered away; it's the operating condition. So the real question isn't "will it be wrong" but "what happens when it is" — and the answer has to be survivable, ideally boring.

The best-fit problems have a cheap, obvious, reversible failure. A drafted email that's off gets edited before it sends. A mis-ranked search result gets scrolled past. A wrongly-tagged ticket gets re-tagged. In each case a human is already in the loop, the error surfaces immediately, and the cost of correcting it is trivial. The AI doesn't have to be right; it has to be right often enough to save time net of the corrections.

The problems to approach with caution are the mirror image: an error that's expensive, irreversible, or invisible until it's done damage. That doesn't mean never — it means the human keeps the final decision, and the AI's job shrinks to making that decision faster, not making it for them. Plenty of high-stakes workflows are perfect for AI as long as you're honest that the model drafts and a person commits.

Does the arithmetic clear?

The third question is the one people skip because it's unromantic: does the math work? Every AI feature has a per-use cost — tokens, latency, the engineering to keep it reliable — and a per-use value. Multiply value by volume, subtract cost, and if the number isn't comfortably positive, nothing about the model's quality will save it.

This kills two kinds of project. The high-value, low-volume kind, where each answer is worth a lot but you only need three a month — often cheaper to have a person do it. And the low-value, high-volume kind, where the per-use benefit is real but so thin that the inference cost eats it alive at scale. The winners sit where meaningful value meets real volume, and the only way to know is to actually estimate it before you build — which almost nobody does, and which I've written about at length in the cost model nobody builds until it's too late.

The shape of a good problem

Put the three questions together and a shape emerges. The AI problems that earn their keep tend to be high-frequency, human-in-the-loop, tolerant of imperfect answers, and measurable within minutes. They're rarely the moonshot in the board deck. They're the repetitive, judgment-light, high-volume task that a team already does by hand and quietly resents.

That's not a counsel of timidity. It's how you build the muscle — and the evaluation infrastructure, and the organizational trust — that makes the ambitious second project possible. Teams that start with a boring, measurable win earn the right to attempt the hard thing. Teams that open with the moonshot usually spend their credibility before they have any.

Start where it's boring

A lot of the strategy work I care about is exactly this triage: taking a list of things a team hopes AI can do and sorting it by feedback loop, cost of error, and unit economics — not by how exciting it sounds. The output is often deflating in the best way. The flashiest idea gets parked, and a task nobody thought to mention becomes the first thing worth shipping, precisely because it clears all three tests.

If you've got a backlog of AI ideas and no confident way to rank them, that ranking is the most valuable hour you can spend — including the parts you think are too small to mention.