A reference architecture for production RAG
The stages every production RAG system needs — indexing, query-time retrieval, and always-on evaluation — and how to order them.
Contents
Short answer: a production RAG system has three parts — an offline indexing pipeline (parse → chunk → embed → store), an online query pipeline (rewrite → hybrid retrieve → rerank → assemble → generate), and always-on evaluation + logging. Order the query stages so recall-wideners come first and precision-tighteners come last, and measure every stage.
This is the hub for everything else I write about RAG. Each box below is a decision with its own deep-dive.
The whole system in one view
1. Indexing (offline)
Runs when documents change, not per query.
- Parse + chunk — extract clean text, split with a structure-aware (recursive) splitter. This decides what's findable. (Chunk size guide.)
- Embed — turn chunks into vectors with your embedding model.
- Store — a vector database with payload metadata for filtering.
Get chunking right here and every downstream stage gets easier.
2. Query (online)
Runs per request, and latency matters.
- Rewrite / contextualize — turn a raw (possibly multi-turn) question into a clean standalone query.
- Hybrid retrieve — dense + sparse in parallel so both meaning and exact terms are covered. (Hybrid search with RRF.)
- Rerank — a cross-encoder scores the top candidates for precision (retrieve wide, rerank to 3–5).
- Assemble + generate — build the final prompt and call the model.
The ordering principle: recall-widening stages early (rewrite, hybrid), precision-tightening stages late (rerank, top-k). Widen the net, then tighten it.
3. Always-on: evaluation + logging
The part teams skip — and then fly blind.
- Log the assembled prompt on every request. It's the single highest-value debug signal; if retrieval or truncation went wrong, you see it immediately.
- Track quality: recall@k and nDCG for retrieval, faithfulness for generation.
- Track cost + latency per config, so quality improvements don't quietly blow your budget.
Which knob to turn first
When quality is bad, go cheapest-first: chunking → hybrid fusion → top-k → reranking. Rerankers are powerful but can dominate your latency budget, so earn the cheap wins before paying for them. And remember most failures are retrieval, not the model.
The rule
Index well offline, order query stages recall-first then precision-last, and never ship retrieval without evaluation and prompt logging.
Related: why RAG fails in production, how to debug a RAG pipeline, and why I don't use LangChain.
Frequently asked
- What does a production RAG architecture look like?
- Three parts: an offline indexing pipeline (parse → chunk → embed → store), an online query pipeline (rewrite → hybrid retrieve → rerank → assemble → generate), and always-on evaluation/observability. Recall-widening stages come early, precision-tightening stages late.
- What order should RAG retrieval stages go in?
- Contextualize/rewrite the query, retrieve with hybrid search, fuse, rerank, then tune top-k — recall first, precision last. Add cheap stages (chunking, hybrid, top-k) before expensive ones (reranking).
- What's the most overlooked part of a RAG architecture?
- Evaluation and logging. Most teams ship retrieval with no way to measure it. Log the assembled prompt on every request and track recall@k, faithfulness, latency, and cost continuously.
Keep reading
Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadWhen NOT to use RAG (RAG vs fine-tuning vs prompting)
RAG became the default reflex for every LLM problem. Often it's the wrong tool. A straight decision framework for RAG vs fine-tuning vs just prompting.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.