When NOT to use RAG (RAG vs fine-tuning vs prompting)
RAG became the default reflex for every LLM problem. Often it's the wrong tool. A straight decision framework for RAG vs fine-tuning vs just prompting.
Contents
Short answer: use RAG only when the problem is missing or changing knowledge. If the problem is behavior (format, tone, a skill the model lacks), that's fine-tuning. If the context is small and stable, just put it in the prompt. RAG became everyone's reflex — and it's often the wrong, most expensive tool for the job.
I build RAG systems for a living, so this may sound odd coming from me: most teams reach for RAG too fast. They add a vector database to a problem that a three-line prompt change would have solved, then spend weeks debugging retrieval they never needed.
The decision, in one picture
Three different problems, three different tools:
- Missing / changing knowledge → RAG. Facts the model doesn't have, that change over time, or that you must cite. This is RAG's actual job.
- New behavior, format, or style → fine-tuning. The model can see the input but can't produce the output you want — a consistent JSON shape, a domain tone, a skill it lacks. No amount of retrieval fixes that.
- Small, stable context → just prompt. If the knowledge fits comfortably in the context window and rarely changes, put it in the prompt. You've just saved yourself an entire subsystem.
The mistake: using RAG for behavior problems
The most common misfire is bolting RAG onto a problem that's really about behavior. Symptoms:
- "The answers are the right facts but the wrong format." → That's fine-tuning (or better prompting), not retrieval.
- "It won't follow our tone." → Behavior. RAG can't teach style.
- "We retrieve the right docs and it still answers badly." → Your problem is downstream of retrieval; adding more retrieval won't help.
RAG can only change what the model knows, never how it behaves.
The other mistake: RAG when a prompt would do
Before standing up embeddings, a vector DB, chunking, and reranking, ask: does the knowledge even fit in the prompt? If your entire corpus is a few dozen pages that change monthly, a long-context prompt is simpler, faster to ship, and easier to debug. RAG earns its complexity when the corpus is large, fast-changing, or needs citations and cost control — not before.
When RAG is right
To be clear — RAG is the correct tool a lot of the time:
- Large or frequently-updated knowledge bases
- Answers that must cite sources
- Per-user or per-tenant private data
- Cost control (retrieve a little context instead of stuffing everything)
And you can combine tools: fine-tune for behavior, RAG for fresh facts. They're complementary — just don't use either as a reflex.
The rule
RAG adds knowledge. Fine-tuning adds behavior. Prompting is free. Diagnose which problem you actually have before you reach for the most complex tool — which, suspiciously often, is the one everyone reaches for first.
Related: how to choose chunk size, hybrid search for RAG, and why I don't use LangChain in production.
Frequently asked
- When should I NOT use RAG?
- Don't use RAG when the problem is behavior rather than knowledge (formatting, tone, a skill the model lacks) — that's a fine-tuning job — or when the needed context is small and stable enough to fit in the prompt. RAG solves missing or changing knowledge, nothing else.
- What's the difference between RAG and fine-tuning?
- RAG adds knowledge at query time by retrieving documents into the prompt. Fine-tuning changes the model's behavior by training on examples. RAG is for facts that change; fine-tuning is for skills, formats, or styles the base model can't produce.
- Is RAG or a long context window better?
- If your whole knowledge base fits comfortably in context and doesn't change often, just put it in the prompt — it's simpler. RAG wins when the corpus is large, changes frequently, or you need citations and cost control.
- Can I combine RAG and fine-tuning?
- Yes. They're complementary: fine-tune for behavior/format, use RAG for up-to-date facts. Just don't reach for either when a better prompt would do the job.
Keep reading
Why I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadA reference architecture for production RAG
The stages every production RAG system needs — indexing, query-time retrieval, and always-on evaluation — and how to order them.
ReadWhy RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.