OCR Confidence Scores: Using Them to Gate What Enters Your RAG Index

Short answer: Treat OCR confidence as a first-class metadata field, not a throwaway log line. Aggregate the per-recognition scores your engine already returns up to the line, block, and page level, then gate before embedding: drop spans below a calibrated threshold, route borderline pages to re-OCR or human review, and store the surviving scores so downstream answers can be flagged as low-trust. The goal is simple — keep garbled text out of the index, because it retrieves as confidently as clean text and drives hallucinations.

Where confidence comes from

Both major OCR paths expose per-recognition confidence you can gate on. Google Cloud Vision returns a confidence on symbols, words, paragraphs, and blocks. Tesseract returns per-word confidence (0–100) via image_to_data — the conf column, with -1 marking non-text rows you should discard before aggregating.

That per-word signal is the raw material. What you actually gate on is an aggregate:

• Line/block confidence: the mean (or a low percentile) of word confidences in that span. A percentile is stricter — one garbage word tanks the score, which is usually what you want.
• Page confidence: the fraction of words above your word-level floor, not the raw mean. "92% of words cleared 60" tells you more about usability than an average dragged around by a few outliers.
• Coverage checks: flag pages where OCR returned far less text than the page area implies. Absent text is as dangerous as wrong text.

Policies: drop, re-OCR, or review

Once you have a score per span, pick a policy per tier. In practice you want all three, applied by confidence band:

• Drop below threshold. Spans under your floor never get embedded. This is the cheap, high-value move — most gains come from just deleting the worst 2–5%.
• Re-OCR at higher DPI. Borderline pages often recover with better input: 300+ DPI rasterization, deskewing, binarization, or a second engine. Scanned PDFs re-rendered at higher resolution are the classic win here.
• Route to human review. Reserve this for high-value, still-bad-after-re-OCR pages — contracts, tables, anything where a wrong number is expensive.

For on-device or app-embedded OCR, do cleanup before the gate: language correction and a custom word list (product names, domain jargon, part numbers) measurably improve output, so fewer good spans get wrongly dropped for looking like gibberish.

Why bother at all: parsing research on RAG shows poor extraction produces duplicated or absent text, and both failure modes raise hallucination risk — the model confidently answers from text that was never really on the page.

Store confidence as metadata

Gating decides what enters. Metadata decides how much you trust what got in. Attach the score to every chunk so retrieval and generation can reason about it:

{
  "text": "Total due: $4,120.00",
  "source": "invoice_2024_q3.pdf",
  "page": 3,
  "ocr_confidence": 0.71,
  "ocr_engine": "tesseract",
  "review_status": "auto"
}

This buys you three things: you can filter at query time (ocr_confidence >= 0.85 for high-stakes questions), surface a "low-confidence source" warning in the UI when a cited chunk is shaky, and re-run just the weak chunks later when you improve the pipeline — no full re-index.

Measuring the lift

Don't guess at thresholds — measure. Build a small labeled eval set (50–100 questions with known answers over your documents) and run it twice: once with everything indexed, once with the confidence gate on.

Watch three things:

• Answer accuracy / faithfulness — should rise as garbage stops surfacing.
• Retrieval hit rate — make sure the gate isn't discarding good text; if it drops, your threshold is too aggressive or your OCR needs the cleanup pass, not a harsher cutoff.
• Coverage — how many documents still have usable content after gating.

Sweep the threshold across a few values and plot accuracy against retained-text volume. There is no universal number; the right cutoff is wherever accuracy plateaus before you start losing recall.

Treat OCR confidence like any other quality signal: gate on it, store it, and prove the lift with an eval set — don't let text you can't read become answers you can't trust.

Related: more writing.