Getting text out of documents that resist

A PDF is a layout, not a document

A PDF stores glyphs at coordinates. It does not store paragraphs, reading order, or the fact that this line is a heading. Every text extractor is reconstructing structure that was thrown away when the file was made, and the reconstruction fails in predictable places.

The four failures you will meet in the first week:

Two columns become interleaved. A naive extractor reads left to right across the page, so a line from column one is followed by a line from column two. The output is grammatical nonsense that no chunker can rescue. Layout-aware extraction fixes it; plain text extraction does not.

Tables become word soup. Cells are just text boxes. Without column detection you get every cell in reading order and no indication of which value belongs to which heading. A financial table extracted this way is worse than useless, because the numbers survive and their meaning does not.

Headers and footers interleave with the body. “Confidential: page 14 of 82” lands in the middle of a sentence, every page, and then gets embedded as part of a chunk.

It is not text at all. A scanned page is an image. Extraction returns an empty string, silently, and a document quietly enters your index with nothing in it. Always assert on characters extracted per page and alert when it is near zero.

The free tool chain, in the order to try it

1. pdfplumber: good text and good table detection for digital PDFs. Slow on large files and worth every second.
2. PyMuPDF: much faster, good layout information. Note its licence is AGPL, which matters if you are shipping a closed product; check before you build on it.
3. Docling or unstructured: higher-level pipelines that handle layout, tables, headings and multiple formats. More opinionated, more dependencies, far less code to write.
4. ocrmypdf wrapping Tesseract: for scans. It adds a text layer to the PDF so the rest of your pipeline works unchanged, which is a better shape than a separate OCR branch.

The paid options (Azure Document Intelligence, AWS Textract, Google Document AI, Adobe’s extraction API) are meaningfully better on hard layouts and cost per page. The honest position: for clean digital PDFs the free chain is competitive; for dense forms, handwriting, and multi-column scans the paid services are still ahead, and if your corpus is 300 documents you can do the hard ones by hand for less than the integration cost either way.

OCR quality is not uniform, and this matters

Tesseract’s accuracy on clean printed English is excellent. On Devanagari, Bengali, Tamil, Arabic and Amharic it is substantially worse, and on handwriting it is poor in every script. Layout complexity compounds it.

Measure before you trust it. Take twenty pages representative of your corpus, type the correct text for a few paragraphs by hand, and compute character error rate. Below about 2% you can proceed. Above 10% the retrieval downstream will fail in ways you will misdiagnose as an embedding problem for weeks, because misrecognised words embed to the wrong place and there is no error message anywhere.

Serialise tables as lines, not as grids

A table has to become text. The instinct is to render it as an ASCII grid or a markdown table, and that is usually the wrong choice for a chunked pipeline for one concrete reason: the header row and the data rows get separated when the chunker splits, and a row of numbers with no headings is unrecoverable.

The more robust serialisation repeats the key with every value:

Region: North · Quarter: Q3 2026 · Revenue: 4,120,000 INR · Growth: 3.1%
Region: South · Quarter: Q3 2026 · Revenue: 2,880,000 INR · Growth: -0.4%

Each line stands alone, survives any split, and embeds meaningfully: a query about southern revenue matches the second line directly. It costs more tokens than a grid, which is a real trade, and it is worth it wherever the table is what people ask about.

For a wide table read as a whole rather than by row, a compact grid inside a single chunk is fine. The rule is: never let a split separate the values from their headings.

Other formats, briefly

HTML: strip navigation and boilerplate before anything else, using the statistical method from the previous module. trafilatura and readability-lxml do main-content extraction well and free.

Office documents: python-docx and openpyxl read structure directly, which is far better than converting to PDF first. Converting a .docx to PDF to extract text is throwing away the structure you wanted, then trying to reconstruct it.

Slides: each slide is a chunk, with the deck title and slide number attached. Speaker notes are often the only prose in the file and are frequently forgotten.

Email: quoted history repeats the same text down a thread. Strip quoted blocks or you will index the same paragraph eleven times, with the duplication effects from the previous module.

Store the parse, not just the text

Keep, for every document: the extracted text, the extractor and version that produced it, the page or offset each chunk came from, and the character count per page. It is a few extra columns.

What it buys you is the ability to answer, eight months later, the question that is otherwise unanswerable: did this document parse correctly? Without it, a corpus of 200,000 documents contains an unknown number of empty and scrambled ones and there is no way to find them except complaints.