Designing an Idempotent Document Ingestion Pipeline (Content Hashing, DLQs, Reprocessing)

Short answer: Make it idempotent by keying every stage on a content hash. Compute a SHA-256 over the raw bytes, derive your chunk and vector IDs from it, and turn every write into a deterministic upsert — so re-running the same input produces the exact same index state instead of duplicating or corrupting it. Everything else (DLQs, retries, reprocessing) is about surviving failure without losing that guarantee.

I've rebuilt an ingestion pipeline twice now, and both times the bug that hurt most wasn't retrieval quality — it was a re-run that silently doubled every chunk in the index. Idempotency isn't a nice-to-have; it's the property that lets you re-run anything, anytime, without fear.

The six stages

A document ingestion pipeline is six stages, and each one can fail independently:

• Source monitoring — poll or webhook off S3, Drive, a CMS, etc.
• Parsing — extract text/structure from PDFs, HTML, docx.
• Chunking — split into retrieval units.
• Metadata — attach source, ACLs, timestamps, version.
• Embedding — vectorize each chunk.
• Indexing — upsert into the vector store.

If any stage is non-deterministic or non-idempotent, a re-run corrupts everything downstream of it. So the design goal is: the same input bytes always yield the same IDs and the same writes.

Content hashing is the whole trick

Compute SHA-256(raw_bytes) at ingest. That hash does three jobs at once:

• Dedupe uploads — two users upload the same PDF, same hash, you process it once.
• Change detection — each sync cycle, compare the stored hash. Unchanged? Skip the entire pipeline. It's a no-op.
• Stable IDs — derive chunk IDs as sha256(doc_hash + chunk_index). Now indexing is an upsert, not an append. Re-running overwrites in place.

Store the hash alongside a timestamp and a pipeline version per document. The version matters: when you change your chunking or embedding model, bump it, and the version becomes part of the skip decision so you can force a clean reprocess without touching source files.

doc_hash = hashlib.sha256(raw_bytes).hexdigest()
prior = index_meta.get(doc_id)
if prior and prior.hash == doc_hash and prior.pipeline_version == PIPELINE_VERSION:
    return  # idempotent skip — nothing changed
chunk_id = hashlib.sha256(f"{doc_hash}:{i}".encode()).hexdigest()
vector_store.upsert(id=chunk_id, ...)  # overwrite, never append
# then delete stale chunks: any chunk_id under this doc not in the new set

The one gotcha: on reprocess, you must delete stale chunks. If a new version has 8 chunks where the old had 12, upserting 8 leaves 4 orphans. Track chunk IDs per document and delete the diff.

Failure isolation: DLQs over silent drops

Run ingestion async, off a queue. Synchronous ingestion couples your failure domains and gives you nowhere to enforce rate limits or cost budgets.

When a stage fails — a corrupt PDF, an embedding API 500 — do not drop the document. Route it to a dead-letter queue. A DLQ captures failed docs instead of losing them, and gives you three things: visibility into what's stuck, a place to inspect the actual failure, and a target for automated reprocessing once the cause is fixed.

The rest of the resilience toolkit:

• Retries with exponential backoff for transient errors (429s, timeouts). Cap the attempts, then DLQ.
• Circuit breakers on the embedding provider — stop hammering a down API and shedding budget.
• Graceful degradation — if metadata enrichment fails but text is fine, index with partial metadata rather than dropping the whole doc.

Because IDs are hash-derived, replaying a DLQ item is safe by construction. Reprocessing is just re-running the pipeline for one doc; idempotency means it can't double-write.

Deterministic reprocessing and a CI gate

Two production requirements turn a demo pipeline into a real one:

• Deterministic reprocessing on source change. A file changes → new hash → the pipeline reprocesses exactly that document and nothing else. No full re-index, no manual intervention.
• ACL propagation. Permissions live in metadata and must re-sync when they change at the source, or you leak documents into retrieval for users who shouldn't see them.

Then gate it. Keep a small labeled eval set and run a retrieval-quality check in CI on every pipeline change — recall@k and a few known query→doc assertions. A chunking tweak that quietly tanks recall should fail the build, not surface in production. Pair that with explicit cost and latency budgets per document so a pathological 400-page PDF can't blow your embedding bill.

Hash first, upsert always, DLQ on failure — then re-running your pipeline is boring, which is exactly what you want.

Related: more writing.