RAG chunking strategies: a practical reference
The common chunking approaches for retrieval, when each one helps, and the tradeoffs — a quick reference you can come back to.
Contents
A "general / reference" post: broad-coverage, useful, and searchable — the SEO surface layer. It doesn't need your personal voice the way an essay does, but it should still be accurate and genuinely helpful. Expand it with your real numbers.
Chunking decides what your retriever can find. Get it wrong and even a perfect embedding model returns junk. Here are the main strategies and when each fits.
Fixed-size chunking
Split text every N tokens with some overlap.
- Good for: uniform documents, a fast first version.
- Weak at: it cuts sentences and ideas in half, which hurts retrieval quality.
Recursive / structure-aware chunking
Split along natural boundaries (headings, paragraphs, sentences) before falling back to size limits.
- Good for: most real documents. Usually the best default.
- Weak at: needs clean structure to shine.
Semantic chunking
Group sentences by embedding similarity so each chunk is one coherent idea.
- Good for: dense, mixed-topic documents.
- Weak at: slower and more expensive to build.
A quick decision guide
| Situation | Start with |
|---|---|
| First version, uniform docs | Fixed-size + overlap |
| Real-world mixed documents | Recursive / structure-aware |
| Dense, high-stakes retrieval | Semantic |
The honest rule: measure retrieval quality, don't guess. Chunking is the cheapest lever to test and the one most people never actually evaluate.
Frequently asked
- What is the best chunking strategy for RAG?
- For most real-world documents, recursive (structure-aware) chunking is the best default — it splits along headings, paragraphs, and sentences before falling back to a size limit, which keeps ideas intact. Fixed-size chunking is fine for a first version; semantic chunking wins for dense, mixed-topic documents.
- How big should RAG chunks be?
- There is no universal number — it depends on your documents and embedding model. Start with structure-aware chunks with some overlap, then measure retrieval quality and adjust. Chunk size is the cheapest lever to test, so tune it empirically rather than guessing.
- Does chunking really affect RAG accuracy?
- Yes, significantly. Chunking decides what your retriever can find, so poor boundaries hurt retrieval even with a perfect embedding model. One published comparison found semantic chunking lifting accuracy to about 71% versus a fixed-size baseline on the same dataset.
Keep reading
Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadWhen NOT to use RAG (RAG vs fine-tuning vs prompting)
RAG became the default reflex for every LLM problem. Often it's the wrong tool. A straight decision framework for RAG vs fine-tuning vs just prompting.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.