Ingestion: PDFs, tables and scans, where most retrieval actually fails

Where the failure is

Teams spend weeks comparing embedding models while the text in their index is broken. Before touching a model, pull twenty chunks at random from the index and read them. If a chunk starts mid-sentence, contains “Page 4 of 12 Confidential” in the middle, or is a run of numbers with no indication of what they measure, no retrieval technique downstream will recover what was lost here. This lesson is about the unglamorous work that decides everything.

A PDF is not text

A PDF is a set of drawing instructions: put this glyph at this coordinate. There is no paragraph, no reading order, often no space character. A text extractor reconstructs all of that by guessing from positions, and it guesses wrong in predictable ways.

Two-column layouts come out interleaved, one line from the left column and one from the right, producing sentences that no human wrote. Headers and footers repeat on every page and land inside every chunk, so the string “Confidential: Internal Use Only” becomes the most common phrase in your corpus and pollutes every embedding. Words hyphenated across a line break arrive as two fragments. Ligatures such as “fi” arrive as a single unusual character that keyword search will never match.

The free tools are good. pymupdf is fast and handles most layouts; pdfplumber gives you positions and is the better choice for tables; pdfminer is the old reliable. For layout-aware extraction (reading order, headings, tables as tables) docling from IBM and marker are free and markedly better than plain extraction on real documents. Strip repeated headers by finding lines that occur on more than half the pages at the same position, and drop them.

Tables lose their heads

A table flattened row by row is the single most damaging thing a naive pipeline does. Row forty of a pricing table becomes the chunk “Pro 50 100 Yes 24h”, and neither an embedding nor a reader knows that 50 is users, 100 is projects and 24h is support response. The column headers were in the chunk three hundred rows earlier.

Convert tables to a form that carries the headers with every row. Markdown tables work for short ones. For long ones, emit one line per row in “header: value” form:

Plan: Pro. Users: 50. Projects: 100. SSO: Yes. Support response: 24h.

For large tables of real data (a price list with thousands of rows) do not embed them at all. Load them into a database and give the model a tool that queries it, which the tool-use module already taught you how to build. A model reading a table row by row is the wrong instrument for a lookup.

Scans and OCR

A scanned document has no text layer, and the only route in is optical character recognition. Tesseract is free and adequate on clean, printed English; it ships trained data for Hindi, Tamil, Bengali and most Indian scripts, with noticeably lower accuracy than on Latin text. PaddleOCR is a stronger free option on complex layouts. Hosted document services from Google, Microsoft and Mistral cost in the region of a dollar or two per thousand pages and are better on poor scans, handwriting and mixed scripts.

OCR errors are not cosmetic. A keyword search fails completely on “invoicc”, and an embedding of a paragraph with a five per cent character error rate drifts measurably. Keep the OCR confidence score per page, flag pages below a threshold for human review, and store a reference to the page image so an answer can show the original rather than the transcription.

The heading a chunk forgot

A chunk that reads “The limit is 500 per month” is useless without knowing it sits under “Enterprise plan” in “API rate limits”. Chunking cut it from its heading path. Prepend the path (document title, then each heading level down to the chunk) to the chunk text before embedding. It costs twenty tokens per chunk and reliably improves recall, because the chunk now says what it is about.

A stronger version has a model write one sentence of context per chunk, describing where it sits in the document, and prepends that instead. Anthropic reported a large reduction in failed retrievals from this combined with keyword search; the cost is one model call per chunk at ingestion, which on a million chunks is real money and real time, so measure the gain on your own set before committing to it.

Other formats, briefly

HTML needs its navigation, cookie banners and footers removed; trafilatura and readability-style extractors do this. Word documents go through python-docx or pandoc. Spreadsheets are records, not prose; treat them like the large-table case. Code should be chunked on function and class boundaries, which tree-sitter finds for any language, not on line counts.

Duplicates and provenance

Near-duplicates (the same policy pasted into four documents) survive into the index and crowd out variety in results. MinHash over shingles at ingestion catches them cheaply.

Every chunk must carry its provenance: source document id, page, character offset, heading path, ingestion time and parser version. That is what lets an answer cite a page, lets you re-parse only the documents a parser bug affected, and lets you re-embed when the model changes. Provenance you did not store at ingestion cannot be reconstructed later.

The eyeball test, again

After every parser change, sample twenty chunks and read them. It takes ten minutes and it catches what no metric does.