Chunk Size and Overlap for RAG: A Reference with Sensible Defaults

Short answer: Split on structure first (headings, table rows, code functions), not on a fixed number. When you genuinely must pick a size, start at ~256-512 tokens with 10-20% overlap for prose, then tune against a small eval set. Tables, code, and transcripts each need different handling, so don't apply one number everywhere.

Answer: structure first, size second

The consistent recommendation across major RAG guides is to order your decisions: structure-first, size-second. A chunk should ideally be a self-contained unit of meaning, a section under a heading, a full table, a single function, before it's a count of tokens.

Only when a structural unit is too large for your embedding model or dilutes retrieval do you fall back to a fixed size. For prose, 256-512 tokens with 10-20% overlap is a defensible default. It's a starting point, not an answer.

Defaults by content type

Tabular data is the classic trap: a fixed token splitter will cut a table mid-row and strip the header, so retrieved rows lose the column names that give them meaning. Keep tables whole, or split by row groups and repeat the header in every chunk.

Why overlap exists, and when it wastes tokens

Overlap copies the last N tokens of one chunk onto the front of the next. It exists so that a sentence or fact straddling a boundary still appears complete in at least one chunk, preserving context across the cut.

The cost is real. Overlap inflates your index size and embedding/storage cost roughly in proportion to the overlap fraction, and it causes near-duplicate chunks to surface together in retrieval, wasting slots in your top-k.

• Use overlap when chunks are size-cut through continuous prose or transcripts.
• Skip overlap when chunks are already self-contained: tables, code functions, Q&A pairs. Overlap there is pure waste.
• 10-20% is the usual band. Above ~25% you're mostly paying to store duplicates.

Interaction with context window and top-k

Chunk size doesn't live alone. Three numbers move together:

• Embedding model context window. Your chunk must fit the embedder's max input (often 512-8192 tokens). Smaller than the limit is usually better for retrieval precision, a huge chunk averages many topics into one blurry vector.
• Retrieval top-k. Smaller chunks mean you need a higher k to gather the same context; larger chunks mean fewer, coarser hits. Budget chunk_size × top_k against your generator's context window and leave room for the prompt and answer.
• Precision vs. recall. Small chunks = precise but fragmented; large chunks = complete but noisy. Tune the pair together, not in isolation.

How to measure and tune

Don't argue about defaults, measure. Build a small eval set of 30-50 real questions with known correct source passages, then sweep a few configurations.

# Sweep chunk size × overlap against a labeled eval set
for chunk_size in [256, 512, 1024]:
    for overlap in [0.0, 0.1, 0.2]:
        index = build_index(docs, chunk_size, overlap)
        hits = sum(
            has_gold_passage(index.search(q.text, top_k=5), q.gold)
            for q in eval_set
        )
        print(chunk_size, overlap,
              "hit_rate:", hits / len(eval_set))

Track two metrics: retrieval hit-rate (did the correct passage make top-k?) and end-to-end answer accuracy (did the LLM answer correctly?). Optimize hit-rate first, it caps everything downstream, then confirm the answer quality. Pick the smallest chunk size that hits your recall target, since smaller is cheaper and more precise.

Bottom line: chunk on structure, default to ~256-512 tokens with 10-20% overlap only when you must, and let a small eval set pick the final numbers.

Related: more writing.