The LLM does the reading. Deterministic code owns the truth.

Modr's Diagram Intelligence does something that reads like magic in a demo: a mechanic photographs a wiring schematic or an exploded parts diagram — the kind of dense, unlabeled page buried in a service manual — and gets back an interactive overlay. Every component is boxed and clickable. There are step-by-step diagnostics tailored to the exact vehicle. And each part links out to somewhere you can actually buy it.

The demo took an afternoon. A single multimodal call to Gemini will happily return components, bounding boxes, instructions, and part explanations, and the first time you see it work on a real schematic it feels done. It is not done. The distance between "the model returned something plausible" and "a mechanic can trust this on a lift with a customer's car in pieces" is the entire product — and it's all guardrails.

This is the honest account of how the LLM calls are actually built: the architecture around them, the three places I refuse to trust the model, and the harness that keeps one flaky call from taking down the response.

The shape of the system

Diagram Intelligence lives in its own Express service inside the Modr monorepo, and the AI layer is deliberately split into small, single-purpose files rather than one god-function:

• genai-client.ts — the client factory. It owns the model names, the fallback model, and the observability gateway. Nothing else knows what model it's talking to.
• prompts.ts — every prompt is a named constant here. No prompt strings are inlined at the call site.
• analyze.ts — the orchestration: the actual generateContent calls, the JSON schemas, parsing, validation, and the tiered parts search.
• cv-refine.ts — classical computer vision that runs after the model, with no LLM in it at all.
• types.ts — the Zod schemas that every model response has to survive.

The one design decision that pays off everywhere is treating the prompt as configuration, not as prose. There isn't one "analyze" prompt; there are several, and they say subtly different things about the same task depending on what happens downstream. The baseline prompt asks the model to box a component and its text label, tightly. The CV-refinement prompts ask for the opposite — a "loose ROI centered on the component," explicitly told that "the bounding boxes will be refined later by computer vision." The prompt is written to match the guardrail that follows it. That coupling is the whole trick, and it's the theme of everything below.

Don't trust the model with coordinates

Here's the first place I stopped trusting the model. Gemini is genuinely excellent at the semantic half of the job — this is a starter relay, that callout is part 8260A, this circuit is the one you'd inspect for a no-crank condition. It is unreliable at the metric half: the exact pixel rectangle around that relay. The boxes come back plausible and a few percent off, which is exactly wrong enough that a clickable hotspot lands next to the part instead of on it.

So the harness doesn't ask the model to be precise. It asks for a loose region of interest — coordinates as percentages, roughly centered — and then hands that ROI to deterministic computer vision to tighten. Two refiners exist. The light one, built on sharp, crops the ROI, runs an Otsu threshold to separate foreground ink from background, and returns the tight bounding box of the connected mark — as long as that box still contains the model's original center point. The heavier one runs OpenCV: grayscale, Gaussian blur, Canny edge detection, contour finding, then picks the smallest contour that actually contains the center.

What makes this a guardrail and not just a post-processing step is the fallbacks. If CV finds no suitable contour, if the tightened box would swallow more than 90% of the region (a sign it locked onto the whole diagram, not the part), or if the image extraction throws — the component keeps its original model-provided box and the pipeline continues. A refinement failure degrades one box's precision; it never fails the request. The model does what it's good at; deterministic code owns the pixels; and neither one can take the other down.

Three gates on every response

Every structured call passes through three independent gates before its output is allowed to become a response. Any one of them can catch a bad generation; together they mean a malformed answer surfaces as a clean server error instead of a broken overlay.

• Gate one — schema in the request. The call sets responseMimeType: 'application/json' and passes an explicit responseJsonSchema. That schema is hand-flattened to at most two levels of nesting, because deeply nested schemas are where structured-output support quietly breaks. Coordinates are declared with minimum: 0, maximum: 100. We ask the model to color inside the lines before it ever answers.
• Gate two — defensive parsing. Even with JSON mode on, the text gets stripped of stray markdown fences before JSON.parse. JSON mode is a strong hint, not a contract, and the cost of assuming otherwise is a 500 on a response you could have salvaged.
• Gate three — Zod on the way out. The parsed object is run through DiagramResponseSchema.safeParse. Bounding boxes are re-checked against 0–100 here, in my own code, not just in the model's instructions. On failure I don't half-render — I throw with the exact failing field paths, so the log tells you which field the model got wrong.

Sitting behind all three is a model-fallback guard in the client. The primary is a Gemini 3 Pro preview; the fallback is Flash. A small predicate watches specifically for "model not found" errors — the failure you get when a preview model ID is rotated or retired out from under you — and transparently retries the same call on the fallback. It's cheap insurance against the one deprecation that would otherwise take the whole feature offline with no code change on my side.

The URLs a model invents don't exist

The parts search is where hallucination stops being cosmetic. A wrong sentence in an explanation is a bad answer; a confident link to a product page that has never existed is a broken promise, and mechanics notice immediately. Asking a model to "find where to buy this part" and trusting the URLs it returns is the fastest way to ship a feature that lies.

So the model is never the source of truth for a link. The search call turns on Gemini's Google Search grounding tool, which means the answer is tied to real retrieved pages, and the harness treats the grounding metadata — not the model's prose — as the authoritative list of URLs. When the model returns grounding chunks but no clean JSON, results are built straight from the chunks. Then every candidate URL runs a gauntlet:

• Allowlist. isValidPartsSearchUrl passes only the seven trusted retailer domains (AutoNation, O'Reilly, NAPA, Advance, AutoZone, Pep Boys, RockAuto) and explicitly rejects Google redirect wrappers, search pages, and localhost.
• Redirect resolution. Grounded links arrive as grounding-api-redirect wrappers. Each is followed to its real destination and re-checked that against the allowlist — a redirect is only as trustworthy as where it lands.
• Liveness. Surviving URLs get a real HTTP HEAD request (falling back to GET for servers that refuse HEAD), on a 5-second timeout, run concurrently. Anything that returns a 404 is dropped before the mechanic ever sees it.

By the time a listing appears in the UI it has been grounded, allowlisted, resolved, and pinged. The model got us candidates. Code decided what was real.

The harness around the calls

The last layer isn't about any single call being correct — it's about the system staying up and staying affordable when calls are slow, flaky, or numerous. A diagram with a dozen parts can fan out to a dozen-plus grounded searches, and treating that naïvely gets expensive and fragile fast.

• Failure is isolated, never fatal. Parts searches run through Promise.allSettled with a concurrency cap, not Promise.all. One part's search timing out yields an empty result for that part; it does not reject the batch. The overlay still renders with everything that succeeded.
• Cost is a dial, not a default. The search takes an explicit strategy — one call for all parts, one call per part, or batches of four. That's the cost/latency/quality knob made visible: fewer calls are cheaper and blunter, per-part calls are sharper and pricier, and the caller chooses per context instead of the tradeoff being buried.
• Slow work is a job, not a request. Generating annotated overlay images is slow, so it runs as a background job with a status that moves pending → running → partial → complete, streaming each step's image as it lands and expiring on a TTL. The mechanic sees the analysis immediately and the richer images fill in, rather than staring at a spinner for the slowest thing in the pipeline.
• Every call is measured. The search returns its own wall-clock timing and LLM-call count alongside the results, and all traffic routes through an observability gateway. You cannot manage the cost of a thing you can't see, and LLM cost hides in call count more than in any single call.

None of this is exotic. It's the same discipline you'd apply to any unreliable, metered, high-latency dependency — which is exactly what a hosted LLM is, once you stop treating it like a magic oracle and start treating it like a network service that sometimes lies.

What the guardrails are really for

Strip Diagram Intelligence down and the architecture is one sentence repeated three times: let the model do the reading, and put deterministic code between its output and the user everywhere the output has to be true. Coordinates get tightened by CV. Structure gets enforced by a schema and then re-enforced by Zod. Facts — the purchasable links — get grounded, allowlisted, and HTTP-checked. The model is upstream of all of it and authoritative over none of it.

That's also why the demo was the easy part. A demo only has to be plausible once, and a good model clears that bar on its own. A product has to be trustworthy every time, on schematics nobody tested, with a mechanic's afternoon riding on the answer — and models don't clear that bar alone. The harness is what clears it.

If there's one thing to take from this past the specifics of automotive diagrams: the interesting engineering in an AI feature is almost never the prompt. It's the boundary you draw around the prompt — the list of things you've decided the model is allowed to be wrong about, and the code that catches it every time it is.