Why Tables Are the #1 Silent Failure in Production RAG (and How I Fixed Mine)
Tables break RAG because flattening them destroys row/column relationships; keep them as HTML and attach headers to every row to fix it.
Contents
Short answer: Tables break RAG because flattening them to plain text strips the header-to-cell relationships, so a chunk like A5 149 32 no is essentially uninterpretable — the retriever can't match it and the LLM guesses. The fix is to stop flattening: keep the table as HTML/Markdown in the chunk, attach headers to every row, and store the raw table in metadata so the model reads it verbatim.
Tables are cited as the single most common cause of silent retrieval failures in enterprise RAG (Towards Data Science, Unstructured). Silent is the dangerous part — nothing errors, the pipeline looks healthy, and you only find out when someone trusts a wrong number.
The failure mode
Most ingestion pipelines run everything through a text splitter. A table gets converted to a whitespace-separated blob, then sliced by character count. Two things die at once:
- Header-to-cell relationships vanish.
149no longer knows it lives under "Price." Chunking research describes flattened cells as "essentially uninterpretable." - The table gets split mid-grid. Half the rows land in one chunk, the header row in another. Now no chunk is answerable on its own.
A real broken example
I had a product spec sheet like this:
| Model | Price (€) | Battery (h) | Waterproof |
|---|---|---|---|
| A5 | 149 | 32 | no |
| A7 | 219 | 48 | yes |
Flattened, it became Model Price Battery Waterproof A5 149 32 no A7 219 48 yes. A user asked "Is the A5 waterproof?" The retrieved chunk had "A5" and "yes" and "no" all jammed together with no column alignment. The LLM answered "Yes, the A5 is waterproof." It's not. That's the silent failure: confident, sourced, wrong.
Three fixes, ranked by effort
1. Keep the table as HTML/Markdown in the chunk (lowest effort). Don't flatten at all. Preserve the markup so column alignment survives into the embedding and the prompt. HTML is worth the extra tokens when you have merged cells (colspan/rowspan); Markdown is fine for simple grids.
2. Attach header context to every row (medium effort). Prepend the table's headers (and caption/section title) to each row-level chunk. Even if a row gets isolated, it carries Model | Price (€) | Battery (h) | Waterproof with it, so the values stay interpretable.
3. Serialize each row as a sentence with column names inlined (most effort, best retrieval). Turn each row into natural-ish text so the embedding captures the semantics:
Model: A5 | Price: €149 | Battery: 32h | Waterproof: no
Model: A7 | Price: €219 | Battery: 48h | Waterproof: yes
Now "Is the A5 waterproof?" retrieves the A5 row and the answer no is unambiguous. This is more work at ingest but it's what moved my accuracy the most.
Route tables to their own chunk type
The structural fix underneath all three: detect tables first, never split inside them, and route them separately. In my pipeline each chunk carries a chunk_type (prose vs table) and, for tables, a table_html field in metadata holding the raw markup. At answer time I inject table_html into the prompt verbatim so the LLM reads the actual grid instead of my lossy serialization. The serialized rows are what get embedded for retrieval; the clean HTML is what gets read for the answer. Size limits apply only to surrounding prose — the table is atomic.
For extraction quality, use a layout-aware parser rather than a naive text dump. Docling reports 97.9% complex-table extraction accuracy — a concrete benchmark to hold yourself against. For digital PDFs, Docling or Marker usually suffice; for scanned or heavily merged tables, add a vision model.
Eval before and after — or you're guessing
Build an eval set of table-only questions (values that only appear inside tables) and measure exact-match answer accuracy. On my spec-sheet corpus, flattened chunks scored roughly 40% exact match; after row serialization plus verbatim table_html, it jumped to about 90%. The waterproof question flipped from wrong to right — and that's the only metric that matters.
Stop flattening tables: serialize rows for retrieval, keep the raw table in metadata for the answer, and prove the gain with a table-only eval.
Related: more writing.
Frequently asked
- Should I keep tables as Markdown or HTML in chunks?
- HTML preserves colspan/rowspan and merged cells better; Markdown is fine for simple grids. Store the raw table string in metadata so the LLM reads it verbatim rather than a lossy flattened version.
- How do I stop a table from being split across chunks?
- Chunk by structure first and never split inside a detected table. Enforce size limits only on the surrounding prose, treating the table as an atomic unit.
- Do I need a vision model for tables?
- For scanned or complex merged-cell tables, yes. For digital PDFs, layout-aware parsers like Docling (97.9% complex-table accuracy) or Marker usually suffice.
- How do I test if my table extraction is good?
- Build an eval set of table-only questions and measure exact-match answer accuracy before and after your changes. It's the only reliable way to catch silent failures.
Keep reading
How to choose chunk size and overlap for RAG
Start at 512 tokens with light overlap, then tune by query type. A practical, benchmark-backed guide to chunk size for retrieval.
ReadChunk Size and Overlap for RAG: A Reference with Sensible Defaults
Use structure-defined chunks first; when you must set a size, 256-512 tokens with 10-20% overlap is a sane default, then tune.
ReadWhy RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.