Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
Short answer: when RAG gives a wrong answer, the model is usually not the culprit — retrieval is, about 73% of the time. If the right context never makes it into the prompt, a perfect LLM still answers wrong. So the fastest path to a better RAG system is fixing retrieval, not swapping models.
Teams debug RAG backwards. The answer looks wrong, so they blame the LLM, tweak the prompt, or upgrade the model. Meanwhile the actual bug sits one layer up, in retrieval, untouched.
The failure map
The numbers are consistent across 2026 analyses:
- ~73% of RAG failures are retrieval failures — the right chunk never got returned.
- Naive retrieve-then-generate fails at retrieval ~40% of the time.
- State-of-the-art RAG answers only ~63% of factual questions correctly, versus ~44% for retrieval without advanced techniques.
Read that last pair carefully: the jump from 44% to 63% comes almost entirely from better retrieval — hybrid search, reranking, chunking — not from a smarter model.
The failure modes hiding inside "hallucination"
"It hallucinated" is a diagnosis that hides the real bug. Break it down:
- Wrong document retrieved. You ask about Q3 revenue; the retriever returns the Q2 doc. The LLM answers faithfully — to the wrong context.
- Right document, bad chunk boundary. The answer got split across two chunks and only half was retrieved.
- Retrieved but truncated. The context was found but cut before it reached the model (token limits).
- Genuinely a generation problem. Right context, wrong answer. This is real — but it's the minority case.
Only the last one is the model's fault. The first three are retrieval, and they're where most of your errors live.
What to fix, in order
Because retrieval dominates, prioritise accordingly:
- Chunking — the cheapest lever, and the one most teams never evaluate. (How to choose chunk size.)
- Hybrid search — add keyword retrieval so exact terms stop slipping through. (Hybrid search with RRF.)
- Reranking — precision on the final few chunks.
- Evaluation — measure retrieval separately from generation so you know which layer to fix.
The rule
Before you touch the model or the prompt, prove retrieval returned the right context. ~73% of the time, that's where the bug is.
Related: a reference architecture for production RAG, how to debug a RAG pipeline, and more writing.
Frequently asked
- Where do production RAG systems actually fail?
- Overwhelmingly at retrieval, not generation — industry analysis puts the failure point at retrieval roughly 73% of the time. If the right context never gets retrieved, even a perfect LLM answers wrong.
- Is RAG failing because of the LLM?
- Usually not. Most 'bad answer' bugs trace to the retrieval step returning the wrong or missing context. Fix retrieval before you touch the model or the prompt.
- How often does naive RAG fail at retrieval?
- Naive retrieve-then-generate pipelines fail at the retrieval step around 40% of the time, and state-of-the-art RAG still answers only ~63% of factual questions correctly — so retrieval quality is the main lever.
Keep reading
Naive retrieve-then-generate isn't enough for production
The 5-line RAG baseline is great for a demo and fragile in production. Here's where it breaks and the upgrade path that fixes it.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadA reference architecture for production RAG
The stages every production RAG system needs — indexing, query-time retrieval, and always-on evaluation — and how to order them.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.