RAG that survives production

A retrieval-augmented generation demo is one of the easiest things in AI to build and one of the hardest to keep working. You embed a few hundred documents, wire up a vector search, stuff the top results into a prompt, and within an afternoon it answers questions about your own data. It feels like magic. Then you point it at the real corpus — fifty thousand documents, half of them near-duplicates, some of them wrong, some of them a year out of date — and the magic curdles into confident, well-formatted nonsense.

The thing to internalize early is that when RAG fails, it almost never fails at the model. The model faithfully summarizes whatever you hand it. It fails at retrieval: you handed it the wrong chunks, or too many, or stale ones, or nothing useful at all and it obliged by making something up. Everything worth engineering in RAG lives in getting the right context in front of the model — and in the discipline to feed it nothing when nothing good exists.

Chunking is the whole ballgame

Before a single query runs, you've already made the decision that determines your ceiling: how you split documents into chunks. Split too small and a chunk loses the context that made it meaningful — a paragraph that says "this is not supported in the EU" detached from the feature it refers to. Split too large and every chunk is mostly noise, so the embedding averages out to something bland that matches everything and answers nothing.

The naive approach — fixed 500-token windows with a bit of overlap — is a fine place to start and a terrible place to stop. Better is to chunk along the document's own structure: sections, headings, list items, table rows. A chunk should be the smallest span that still stands on its own, and it should carry its lineage — document title, section path, last-modified date — as metadata, because half the failures you'll debug later come down to a chunk that was technically retrieved but had no way to signal it was three versions obsolete.

Recall first, then precision

A single vector search does one thing well: it finds text that means something similar to the query. It's blind to the query that shares no vocabulary with the answer, and blind to exact terms — part numbers, error codes, proper nouns — where the literal string is the whole point. The fix is not a better embedding model. It's to stop relying on one retrieval method.

export async function retrieve(q: string) {
  // Cast a wide net: two methods that fail differently
  const [dense, sparse] = await Promise.all([
    vectorSearch(q, { k: 50 }),   // semantic similarity
    keywordSearch(q, { k: 50 }),  // BM25 — exact terms, codes, names
  ]);
  const candidates = mergeUnique(dense, sparse);   // ~80 rough matches
  return rerank(q, candidates);                    // precision comes next
}

The mental model is two stages with opposite goals. Stage one — retrieval — optimizes for recall: cast the widest reasonable net, accept that most of it is junk, and make sure the real answer is somewhere in the pool. Stage two — ranking — optimizes for precision: from that noisy pool, surface the handful of chunks that actually deserve to reach the model. Conflating the two is the single most common RAG design error.

The reranker earns its keep

Vector similarity is a cheap approximation of relevance, and it shows: the chunk ranked first by cosine distance is frequently not the most useful one. A cross-encoder reranker — a model that reads the query and each candidate together and scores the pair directly — is dramatically more accurate at judging relevance, because it never has to compress either side into a single vector.

The reranker also lets you send the model less. A common instinct is to stuff twenty chunks into the context "to be safe," but a model handed twenty chunks — eighteen of them irrelevant — gets distracted, pads its answer with tangents, and costs more per call. Six well-ranked chunks beat twenty unranked ones on quality, latency, and price at the same time. The reranker is what makes "send less" safe.

Knowing when to say nothing

Here is the behavior that separates a RAG system you can put in front of customers from one you can't: it declines. When retrieval comes back weak — every candidate scoring below a relevance threshold — the correct output is not a fluent guess. It's "I don't have enough information to answer that," optionally with the closest thing it did find.

const ranked = await retrieve(query);
const strong = ranked.filter(c => c.score >= RELEVANCE_FLOOR);

if (strong.length === 0) {
  return { answer: NO_CONTEXT_RESPONSE, citations: [] };  // refuse, don't improvise
}
return generate({ query, context: strong.slice(0, 6), requireCitations: true });

Pair that with mandatory citation: every claim in the answer must point back to a specific chunk, and an answer that cites nothing is treated as a failure. This does two jobs at once. It gives the user a way to verify, and it gives you a cheap, automatic hallucination check — a sentence that asserts something no retrieved chunk supports is a bug you can detect without a human reading every response.

Freshness and the stale-index trap

The demo is built against a snapshot. Production runs against a corpus that changes — docs get edited, prices update, policies are rewritten — and the index doesn't know until you tell it. A RAG system confidently quoting last quarter's refund policy is worse than no system, because it's wrong with a citation, which reads as authoritative.

So re-indexing can't be a manual quarterly chore. Tie it to the source: when a document changes, its chunks get re-embedded and the old ones retired, ideally through the same event that updates the source of record. Carry a last_modified on every chunk and surface it, so a stale answer at least admits its own age instead of masquerading as current.

Evaluate retrieval on its own

When a RAG answer is wrong, the instinct is to blame the prompt or the model and start fiddling. Usually the fault is upstream, and you can't see it because you're only looking at the final answer. So I measure retrieval as its own component, separate from generation: for a set of real questions with known-relevant chunks, does retrieval actually surface them, and does the reranker put them near the top?

Recall at the retrieval stage and rank of the first relevant chunk after reranking are two numbers that will tell you more about your system's health than any amount of reading transcripts. When the answer is bad, they tell you instantly whether the right context was even in the room. That separation is the same discipline I bring to any system — measure the parts, not just the whole. It's the throughline in how I think about evaluation generally.

The unglamorous conclusion

Nothing in a durable RAG system is exciting. It's structural chunking, hybrid retrieval, a reranker, a refusal threshold, mandatory citations, event-driven freshness, and component-level evals. There's no clever prompt that substitutes for any of it, and no bigger model that rescues bad retrieval. The model was never the problem; it just gets blamed for problems that were decided three steps upstream.

That's the work: taking a RAG prototype that dazzles on curated questions and making it hold up against the messy, changing, adversarial reality of a real corpus. If yours is impressive in the demo and unpredictable in the wild, the fix is almost certainly in the retrieval path, and that's a good place to start looking.