How to reduce RAG hallucinations: a practical checklist
Most RAG hallucinations are retrieval bugs in disguise. A concrete checklist to cut them — grounding, retrieval quality, and faithfulness checks.
Contents
Short answer: most RAG hallucinations are retrieval bugs in disguise — the model answers faithfully to wrong or missing context. Cut them in layers: fix retrieval first, then instruct the model to answer only from context (and abstain otherwise), add citations, and run a faithfulness check. Prompt tricks alone won't fix a retrieval problem.
Before you tune anything, separate the two failure types:
- Right context, made-up answer → true generation hallucination (the minority).
- Wrong/missing context, faithful answer → a retrieval failure wearing a hallucination costume (the majority — ~73% of RAG failures).
Now the checklist, in order of impact.
1. Fix retrieval (biggest win)
If the right context isn't in the prompt, nothing downstream helps.
- Structure-aware chunking so answers aren't split.
- Hybrid search so exact terms aren't missed.
- Reranking for precision on the final chunks.
2. Ground the model explicitly
Tell it, unambiguously, to use only what it was given:
Answer using ONLY the context below. If the context does not contain
the answer, reply exactly: "I don't have enough information."
Do not use outside knowledge.This one instruction turns many silent fabrications into honest abstentions.
3. Let it say "I don't know"
A system that can't abstain will hallucinate on out-of-scope questions. Make "I don't have enough information" a first-class, acceptable answer — and handle zero-result retrieval explicitly instead of forcing an answer from junk.
4. Add citations
Require the model to cite which retrieved chunk each claim came from. Citations make hallucinations visible (a claim with no source is a red flag) and let users verify — they also raise trust.
5. Check faithfulness automatically
Measure whether answers stick to the retrieved context. A faithfulness metric (does every claim trace to context?) catches regressions before users do. Aim high — production targets sit around faithfulness > 0.9.
6. Watch truncation
Right context retrieved but cut off before the model saw it produces "hallucinations" that are really lost data. Log the assembled prompt and confirm the context actually made it in.
The rule
Hallucination is a symptom. Fix retrieval, force grounding, allow "I don't know," and verify faithfulness — in that order.
Related: why RAG fails in production, how to debug a RAG pipeline, and more writing.
Frequently asked
- How do I stop my RAG from hallucinating?
- Attack it in layers: fix retrieval so the right context is present, instruct the model to answer only from context and say 'I don't know' otherwise, add citations, and run a faithfulness check. Most hallucinations are actually retrieval failures, so start there.
- Why does RAG hallucinate even with retrieval?
- Because the retrieved context is often wrong or incomplete — the model then answers faithfully to bad context. Other causes are truncation (right context cut off) and no instruction to abstain when context is missing.
- What's the fastest win against RAG hallucinations?
- Fixing retrieval quality (chunking, hybrid search, reranking). Around 73% of RAG failures are retrieval failures, so better context beats prompt tricks.
Keep reading
Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadA reference architecture for production RAG
The stages every production RAG system needs — indexing, query-time retrieval, and always-on evaluation — and how to order them.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.