Naive retrieve-then-generate isn't enough for production
The 5-line RAG baseline is great for a demo and fragile in production. Here's where it breaks and the upgrade path that fixes it.
Short answer: the 5-line "embed the query, search, stuff results into a prompt" baseline is perfect for a demo and fragile in production, where it fails at retrieval ~40% of the time. The fix isn't a bigger model — it's a better retrieval pipeline: hybrid search, reranking, and query rewriting. That's the difference between ~44% and ~63% factual accuracy.
The baseline everyone starts with
hits = store.search(embed(query), k=5)
context = "\n\n".join(h.text for h in hits)
answer = llm.chat(f"Context:\n{context}\n\nQ: {query}")This is a great starting point. Ship it to learn the shape of the problem. Just don't expect it to survive real users.
Where it breaks
- Exact terms slip through. Dense embeddings blur product codes, acronyms, and function names into fuzzy neighborhoods, so keyword-precise queries miss. (Why dense retrieval fails on exact terms.)
- Ideas get cut across chunks. Fixed-size splitting cuts sentences and tables in half, so the answer is never fully in one retrieved chunk. (Chunk size guide.)
- Follow-up questions collapse. "What about the second one?" embedded alone retrieves nothing useful — multi-turn needs query rewriting.
- No precision on the final few. Top-k by cosine alone puts mediocre chunks in front of the model.
The upgrade path
You don't need all of it on day one. Add in order of ROI:
- Hybrid search (dense + sparse, fused with RRF) → catches exact terms. (How.)
- Structure-aware chunking → stops cutting ideas in half.
- Reranking → precision on the final 3–5 chunks.
- Query rewriting → fixes multi-turn and short queries.
- Evaluation → so you know which change actually helped.
Each is a well-understood technique with a big, measurable payoff — and together they're the gap between a demo and a system. See the production RAG reference architecture for how they fit.
The rule
Naive RAG is a starting line, not a finish line. The accuracy gains live in the retrieval pipeline, not in a bigger model.
Related: why RAG fails in production, production RAG architecture, and more writing.
Frequently asked
- Why isn't basic RAG good enough for production?
- Naive retrieve-then-generate fails at the retrieval step around 40% of the time — it misses exact terms, cuts ideas across chunks, and returns fuzzy matches. Production RAG adds hybrid search, reranking, and query rewriting to close that gap.
- What's wrong with the simple RAG tutorial approach?
- Nothing, for a prototype. It just assumes dense-only retrieval and fixed chunks are good enough, which breaks on rare terms, multi-turn questions, and mixed documents. Each of those has a well-known fix.
- How much better is advanced RAG than naive RAG?
- Straightforward retrieval scores around 44% on factual questions; advanced techniques push state-of-the-art RAG to about 63% — the gain comes from better retrieval, not a better model.
Keep reading
Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadA reference architecture for production RAG
The stages every production RAG system needs — indexing, query-time retrieval, and always-on evaluation — and how to order them.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.