How to choose chunk size and overlap for RAG
Start at 512 tokens with light overlap, then tune by query type. A practical, benchmark-backed guide to chunk size for retrieval.
Contents
Short answer: start at 512 tokens with a recursive splitter and 10–20% overlap, then tune by query type — smaller (256–512) for factoid lookups, larger (1024+ or page-level) for analytical questions. Then confirm the choice against a small eval set on your own data. Don't copy a number from a blog; measure.
Chunking decides what your retriever can even find. Get it wrong and a perfect embedding model still returns the wrong context. Yet most teams pick a chunk size once, by vibes, and never revisit it.
Pick by query type
The single most useful mental model: chunk size trades precision for context.
- Factoid / lookup queries ("what's the refund window?") → 256–512 tokens. Small chunks keep the answer dense and unambiguous, so retrieval is precise.
- Analytical / synthesis queries ("compare our Q2 and Q3 strategy") → 1024+ tokens or page-level. The model needs surrounding context to reason, and tiny fragments starve it.
If your app serves both, that's an argument for hierarchical/parent-document retrieval (retrieve small, return large) rather than a single global size.
Try it yourself
Drag the sliders — shrink the size in fixed mode and watch the mid-sentence cuts climb; switch to recursive and they vanish. This is the whole argument, made playable:
Retrieval-augmented generation grounds a language model in your own data. First you split documents into chunks.
Then you embed each chunk into a vector.
At query time you embed the question, find the nearest chunks, and pass them to the model as context.
The quality of that context decides the quality of the answer. Chunking is the step most teams get wrong.
Split too small and you lose meaning; split too large and retrieval gets fuzzy.
The goal is chunks that each hold one coherent idea.
Edit the source text
Notice: fixed chunking slices at a character count — watch the mid-sentence cuts climb as size shrinks. Recursive respects sentence boundaries, so ideas stay intact. That difference is most of why recursive out-retrieves fixed in practice.
The overlap myth
Overlap is supposed to stop ideas being cut mid-thought. In practice it's oversold. A 2026 systematic analysis found chunk overlap produced no measurable retrieval benefit on many corpora — it just inflated index size and cost.
So: start at 10–20% overlap, but treat it as a hypothesis to test, not a rule. On structured docs with a good recursive splitter, you can often drop it entirely.
What the benchmarks actually say
In a February 2026 comparison of 7 chunking strategies over 50 academic papers, recursive 512-token splitting ranked #1 at ~69% accuracy — beating fancier methods on the cost/accuracy ratio. Semantic chunking can push retrieval accuracy to ~70%, but at meaningfully higher compute. (More on that tradeoff in the semantic-vs-recursive comparison.)
Translation: recursive 512 is a strong default. Reach for semantic chunking only when you have long, narrative documents and evidence it's worth the cost.
Measure it for your corpus
Every "best chunk size" number is corpus-dependent. The reliable move takes an afternoon:
- Build a small labeled set of ~30–50 real queries with the passage that should answer each.
- Sweep a few configs (256/512/1024, overlap on/off).
- Compare recall@k (did the right passage get retrieved?) and end-to-end answer correctness.
- Ship the winner; re-check when your data changes.
Chunking is the cheapest lever in the whole RAG stack to test — and the one most people never actually evaluate.
The rule
Default to recursive 512 with light overlap. Split your sizing by query type. Then let a small eval set — not a blog post — pick the final number.
Related: semantic vs recursive chunking, hybrid search for RAG, and why I don't use LangChain in production.
Frequently asked
- What chunk size should I use for RAG?
- Start with 512 tokens using a recursive splitter, then tune by query type: 256–512 tokens for factoid lookups and 1024+ (or page-level) for analytical, synthesis-heavy questions. Always confirm with a small eval set on your own corpus.
- Do I need chunk overlap?
- Often not. A 2026 systematic analysis found overlap gave no measurable retrieval benefit on many corpora while raising indexing cost. Start at 10–20% and test whether it actually helps yours.
- Is a bigger chunk size always better?
- No. Bigger chunks help synthesis but dilute precision and raise token cost. Match the size to what the query needs, not to a single global setting.
- How do I measure the right chunk size?
- Build a small labeled query→answer set, sweep a few sizes, and compare recall@k and answer correctness. Chunking is the cheapest lever to test empirically.
Keep reading
Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadNaive retrieve-then-generate isn't enough for production
The 5-line RAG baseline is great for a demo and fragile in production. Here's where it breaks and the upgrade path that fixes it.
ReadHybrid search for RAG: combining BM25 and dense vectors with RRF
Why you can't average keyword and vector scores — and how Reciprocal Rank Fusion combines them for an 8–15% accuracy gain.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.