Cleaning Extracted Text Before Embedding: Headers, Footers, Hyphenation, and Noise

Short answer: Clean in a fixed order before chunking: detect and strip repeating headers/footers/page numbers, de-hyphenate words broken across line breaks and rejoin wrapped sentences, then normalize whitespace and unicode. Do this before chunking so boilerplate never gets embedded — but stop short of destroying the table structure and heading markers your chunker relies on.

PDFs are positioned layouts, not structured markup. The file describes where glyphs sit on a page, not what is a heading, a body paragraph, or a running footer. That is why headers, footers, and page numbers leak straight into extracted text, and why parsing research repeatedly finds that poor cleaning produces out-of-order chunks, lost structure, and text that is duplicated or missing entirely. Cleaning is the step that turns raw extraction into something safe to embed.

Detect and remove repeating boilerplate

Headers, footers, and page numbers repeat on nearly every page. Once you chunk, that repetition dominates: the same string appears in hundreds of chunks, skewing embeddings toward the boilerplate instead of the content, and burning context tokens at query time.

The reliable way to find boilerplate is frequency across pages, not pattern matching:

• Extract text per page, keep the top and bottom N lines of each.
• Normalize each candidate line (trim, collapse spaces, replace digit runs with a placeholder so Page 3 and Page 4 collapse to one signature).
• Count how many pages each signature appears in. If a line shows up on more than ~50–70% of pages in the same position, it is boilerplate — drop it.

Use a threshold, not exact match. Page numbers change every page, so Page \d+ of \d+ and running dates only collapse once you mask the variable parts.

De-hyphenate and rejoin wrapped lines

Extraction preserves the visual line breaks of the layout, so you get two distinct problems.

Hyphenation splits a word across lines: inter-\nnational. If you embed that as-is, you get two junk tokens instead of international. Rejoin a trailing hyphen when the next line begins lowercase — but be conservative, because real hyphenated compounds (well-known, state-of-the-art) must survive.

Wrapped sentences are single sentences broken into many short lines. Collapse a line break into a space when the line does not end sentence-terminal punctuation and the next line continues the thought. Keep the break when it marks a genuine paragraph or list boundary.

import re
 
def dehyphenate(text: str) -> str:
    # join hyphen + newline when the continuation starts lowercase
    text = re.sub(r"(\w)-\n(\p{Ll})", r"\1\2", text) \
        if False else re.sub(r"([A-Za-z])-\n([a-z])", r"\1\2", text)
    # collapse soft-wrapped lines: no sentence end, next line continues
    text = re.sub(r"([^\.\?\!:\n])\n(?=[a-z(])", r"\1 ", text)
    return text

Tune the character classes to your corpus; the two rules above catch the common cases without merging list items.

Normalize unicode, ligatures, and control characters

Extraction from font glyphs introduces characters that look fine to a human but fragment tokenization:

• Ligatures: fi and fl are single code points. NFKC normalization decomposes them back to fi/fl.
• Look-alike punctuation: smart quotes, en/em dashes, and non-breaking spaces should be folded to canonical forms so identical text embeds identically.
• Control and zero-width characters: strip \x00–\x1f (except \n/\t), soft hyphens (­), and zero-width spaces (​) — they are invisible noise.
• Whitespace: collapse runs of spaces and tabs, trim trailing spaces, and cap consecutive blank lines at one or two.

Apply unicodedata.normalize("NFKC", text) first, then your targeted replacements.

What NOT to clean

Aggressive cleaning does more damage than dirty text. Do not:

• Flatten tables. Row and column boundaries carry meaning. If you smash a table into prose, the numbers lose their labels and retrieval returns nonsense.
• Delete heading markers. If you chunk on headings (Markdown #, numbered sections), stripping them destroys your split points and your chunk metadata.
• Lowercase everything. Modern embedding models are case-aware; casing distinguishes acronyms, proper nouns, and code. Over-normalizing loses signal.
• Strip all punctuation. Sentence boundaries, list markers, and code symbols are part of the meaning.

The goal is a clean reading order with noise removed and structure intact — not a bag of lowercase words.

Clean boilerplate, hyphens, and unicode before chunking; preserve tables, headings, and case.

Related: more writing.