The single most useful log line in a RAG system
If you log one thing in your RAG pipeline, log the final assembled prompt. It's the fastest path from 'bad answer' to root cause.
Short answer: if you log exactly one thing in your RAG pipeline, log the final assembled prompt — the exact string sent to the model, on every request. It's the fastest path from "the answer is wrong" to "here's why," because almost every RAG bug is visible in it.
I've debugged a lot of RAG systems. The teams that move fast all have one habit in common: they can pull up the exact prompt any given answer was generated from. The teams that flail can't.
Why the prompt, specifically
Everyone logs something — retrieved chunk IDs, scores, latency. Useful, but they describe the pipeline's intent, not what the model actually saw. The assembled prompt is the ground truth. Reading it answers, in one glance:
- Was the right context retrieved? (If it's not in the prompt, retrieval failed.)
- Did it survive truncation? (If it's cut off, that's your bug — not the model.)
- Is the formatting mangled? (Chunks concatenated into noise, missing separators.)
- Where does the answer's mistake come from? (Now you can actually see it.)
The gap between "what retrieval returned" and "what the model received" is exactly where silent failures live — truncation, reordering, template bugs. Only the assembled prompt shows that gap.
How to do it
prompt = build_prompt(system, context, query)
logger.info("llm.prompt", extra={"prompt": prompt, "request_id": rid})
answer = llm.chat(prompt)
logger.info("llm.answer", extra={"answer": answer, "request_id": rid})Tie the prompt and answer together with a request ID. In high-traffic systems, sample (log 1–5%) plus always-log on error. Storage is trivial next to the cost of the LLM call — and the debugging time it saves is enormous.
The framework catch
This is also my main practical gripe with heavy frameworks: they assemble the prompt somewhere you can't easily see. When the highest-value log line is the one the abstraction hides, debugging becomes archaeology. Owning prompt assembly means owning this log — which is a big part of why I don't use LangChain in production.
The rule
Log the exact assembled prompt, tied to a request ID, on every request. It's the cheapest observability you'll ever add and the first thing you'll reach for when something breaks.
Related: how to debug a RAG pipeline, why RAG fails in production, and more writing.
Frequently asked
- What should I log in a RAG system?
- The final assembled prompt sent to the model, on every request. It shows exactly what context the LLM actually received, which is where most RAG bugs reveal themselves.
- Why log the prompt instead of the retrieved chunks?
- Chunks tell you what retrieval returned; the assembled prompt tells you what actually reached the model after truncation, formatting, and ordering. The gap between those two is where silent bugs hide.
- Isn't logging the full prompt expensive?
- It's cheap relative to the LLM call itself, and you can sample in high-traffic systems. The debugging time it saves dwarfs the storage cost.
Keep reading
How to debug a RAG pipeline that returns wrong answers
A step-by-step debug order for bad RAG answers — start with the assembled prompt, not the model. Most bugs are retrieval, not generation.
ReadWhy RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.