{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"documents-that-fight-back","title":"Getting text out of documents that resist","url":"https://addaly.com/learn/context-engineering/documents-that-fight-back","position":21,"of":87,"minutes":10,"answer":"Extraction is reconstruction of structure the file threw away, so expect column interleaving, orphaned table values, boilerplate in the body and silently empty scans: assert on characters per page.","keyIdea":"Extraction is reconstruction of structure the file threw away, so expect column interleaving, orphaned table values, boilerplate in the body and silently empty scans: assert on characters per page, measure OCR error on your own scripts, and serialise tables one self-contained line per row."},"body":{"format":"markdown","text":"## A PDF is a layout, not a document\n\nA PDF stores glyphs at coordinates. It does not store paragraphs, reading order, or the fact that this line is a heading. Every text extractor is reconstructing structure that was thrown away when the file was made, and the reconstruction fails in predictable places.\n\nThe four failures you will meet in the first week:\n\n**Two columns become interleaved.** A naive extractor reads left to right across the page, so a line from column one is followed by a line from column two. The output is grammatical nonsense that no chunker can rescue. Layout-aware extraction fixes it; plain text extraction does not.\n\n**Tables become word soup.** Cells are just text boxes. Without column detection you get every cell in reading order and no indication of which value belongs to which heading. A financial table extracted this way is worse than useless, because the numbers survive and their meaning does not.\n\n**Headers and footers interleave with the body.** \"Confidential: page 14 of 82\" lands in the middle of a sentence, every page, and then gets embedded as part of a chunk.\n\n**It is not text at all.** A scanned page is an image. Extraction returns an empty string, silently, and a document quietly enters your index with nothing in it. Always assert on characters extracted per page and alert when it is near zero.\n\n## The free tool chain, in the order to try it\n\n1. **pdfplumber**: good text and good table detection for digital PDFs. Slow on large files and worth every second.\n2. **PyMuPDF**: much faster, good layout information. Note its licence is AGPL, which matters if you are shipping a closed product; check before you build on it.\n3. **Docling** or **unstructured**: higher-level pipelines that handle layout, tables, headings and multiple formats. More opinionated, more dependencies, far less code to write.\n4. **ocrmypdf** wrapping **Tesseract**: for scans. It adds a text layer to the PDF so the rest of your pipeline works unchanged, which is a better shape than a separate OCR branch.\n\nThe paid options (Azure Document Intelligence, AWS Textract, Google Document AI, Adobe's extraction API) are meaningfully better on hard layouts and cost per page. The honest position: for clean digital PDFs the free chain is competitive; for dense forms, handwriting, and multi-column scans the paid services are still ahead, and if your corpus is 300 documents you can do the hard ones by hand for less than the integration cost either way.\n\n## OCR quality is not uniform, and this matters\n\nTesseract's accuracy on clean printed English is excellent. On Devanagari, Bengali, Tamil, Arabic and Amharic it is substantially worse, and on handwriting it is poor in every script. Layout complexity compounds it.\n\nMeasure before you trust it. Take twenty pages representative of your corpus, type the correct text for a few paragraphs by hand, and compute character error rate. Below about 2% you can proceed. Above 10% the retrieval downstream will fail in ways you will misdiagnose as an embedding problem for weeks, because misrecognised words embed to the wrong place and there is no error message anywhere.\n\n## Serialise tables as lines, not as grids\n\nA table has to become text. The instinct is to render it as an ASCII grid or a markdown table, and that is usually the wrong choice for a chunked pipeline for one concrete reason: the header row and the data rows get separated when the chunker splits, and a row of numbers with no headings is unrecoverable.\n\nThe more robust serialisation repeats the key with every value:\n\n```\nRegion: North · Quarter: Q3 2026 · Revenue: 4,120,000 INR · Growth: 3.1%\nRegion: South · Quarter: Q3 2026 · Revenue: 2,880,000 INR · Growth: -0.4%\n```\n\nEach line stands alone, survives any split, and embeds meaningfully: a query about southern revenue matches the second line directly. It costs more tokens than a grid, which is a real trade, and it is worth it wherever the table is what people ask about.\n\nFor a wide table read as a whole rather than by row, a compact grid inside a single chunk is fine. The rule is: never let a split separate the values from their headings.\n\n## Other formats, briefly\n\n**HTML**: strip navigation and boilerplate before anything else, using the statistical method from the previous module. `trafilatura` and `readability-lxml` do main-content extraction well and free.\n\n**Office documents**: `python-docx` and `openpyxl` read structure directly, which is far better than converting to PDF first. Converting a `.docx` to PDF to extract text is throwing away the structure you wanted, then trying to reconstruct it.\n\n**Slides**: each slide is a chunk, with the deck title and slide number attached. Speaker notes are often the only prose in the file and are frequently forgotten.\n\n**Email**: quoted history repeats the same text down a thread. Strip quoted blocks or you will index the same paragraph eleven times, with the duplication effects from the previous module.\n\n## Store the parse, not just the text\n\nKeep, for every document: the extracted text, the extractor and version that produced it, the page or offset each chunk came from, and the character count per page. It is a few extra columns.\n\nWhat it buys you is the ability to answer, eight months later, the question that is otherwise unanswerable: *did this document parse correctly?* Without it, a corpus of 200,000 documents contains an unknown number of empty and scrambled ones and there is no way to find them except complaints."},"check":{"question":"A financial table is extracted into a markdown grid, then chunked at 400 tokens. Answers about specific figures become wrong in a way that looks like hallucination. What is the mechanism?","options":[{"text":"The chunker split between the header row and the data rows, leaving numbers with no indication of which column they belong to.","correct":true},{"text":"Markdown pipe characters tokenise badly, inflating the token count so fewer rows fit.","correct":false,"tempting":"Punctuation does cost tokens, but a higher token count would truncate rows visibly rather than misattribute values."},{"text":"Numbers embed poorly, so the retriever returns the wrong table entirely.","correct":false,"tempting":"Numeric text does embed weakly, but that would produce answers from an unrelated table rather than plausible wrong values from the right one."},{"text":"The model cannot perform arithmetic reliably, so derived figures are unreliable regardless of extraction.","correct":false,"tempting":"A genuine limitation, but the failure described is in reported values rather than in calculations over them."}]},"prev":{"slug":"saying-what-was-left-out","title":"Admitting the set is partial","url":"https://addaly.com/learn/context-engineering/saying-what-was-left-out"},"next":{"slug":"chunking-without-losing-meaning","title":"Chunking that does not destroy meaning","url":"https://addaly.com/learn/context-engineering/chunking-without-losing-meaning"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Getting text out of documents that resist', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/documents-that-fight-back (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Getting text out of documents that resist","courseTitle":"Context Engineering","module":{"index":3,"slug":"retrieval-as-a-pipeline","title":"Retrieval as a context pipeline"},"answer":null,"keyIdea":"Extraction is reconstruction of structure the file threw away, so expect column interleaving, orphaned table values, boilerplate in the body and silently empty scans: assert on characters per page, measure OCR error on your own scripts, and serialise tables one self-contained line per row.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-pdf-is-a-layout-not-a-document","runs":[{"text":"A PDF is a layout, not a document"}]},{"t":"p","runs":[{"text":"A PDF stores glyphs at coordinates. It does not store paragraphs, reading order, or the fact that this line is a heading. Every text extractor is reconstructing structure that was thrown away when the file was made, and the reconstruction fails in predictable places."}]},{"t":"p","runs":[{"text":"The four failures you will meet in the first week:"}]},{"t":"p","runs":[{"text":"Two columns become interleaved.","b":true},{"text":" A naive extractor reads left to right across the page, so a line from column one is followed by a line from column two. The output is grammatical nonsense that no chunker can rescue. Layout-aware extraction fixes it; plain text extraction does not."}]},{"t":"p","runs":[{"text":"Tables become word soup.","b":true},{"text":" Cells are just text boxes. Without column detection you get every cell in reading order and no indication of which value belongs to which heading. A financial table extracted this way is worse than useless, because the numbers survive and their meaning does not."}]},{"t":"p","runs":[{"text":"Headers and footers interleave with the body.","b":true},{"text":" “Confidential: page 14 of 82” lands in the middle of a sentence, every page, and then gets embedded as part of a chunk."}]},{"t":"p","runs":[{"text":"It is not text at all.","b":true},{"text":" A scanned page is an image. Extraction returns an empty string, silently, and a document quietly enters your index with nothing in it. Always assert on characters extracted per page and alert when it is near zero."}]},{"t":"h2","id":"the-free-tool-chain-in-the-order-to-try-it","runs":[{"text":"The free tool chain, in the order to try it"}]},{"t":"ol","start":1,"items":[[{"text":"pdfplumber","b":true},{"text":": good text and good table detection for digital PDFs. Slow on large files and worth every second."}],[{"text":"PyMuPDF","b":true},{"text":": much faster, good layout information. Note its licence is AGPL, which matters if you are shipping a closed product; check before you build on it."}],[{"text":"Docling","b":true},{"text":" or "},{"text":"unstructured","b":true},{"text":": higher-level pipelines that handle layout, tables, headings and multiple formats. More opinionated, more dependencies, far less code to write."}],[{"text":"ocrmypdf","b":true},{"text":" wrapping "},{"text":"Tesseract","b":true},{"text":": for scans. It adds a text layer to the PDF so the rest of your pipeline works unchanged, which is a better shape than a separate OCR branch."}]]},{"t":"p","runs":[{"text":"The paid options (Azure Document Intelligence, AWS Textract, Google Document AI, Adobe’s extraction API) are meaningfully better on hard layouts and cost per page. The honest position: for clean digital PDFs the free chain is competitive; for dense forms, handwriting, and multi-column scans the paid services are still ahead, and if your corpus is 300 documents you can do the hard ones by hand for less than the integration cost either way."}]},{"t":"h2","id":"ocr-quality-is-not-uniform-and-this-matters","runs":[{"text":"OCR quality is not uniform, and this matters"}]},{"t":"p","runs":[{"text":"Tesseract’s accuracy on clean printed English is excellent. On Devanagari, Bengali, Tamil, Arabic and Amharic it is substantially worse, and on handwriting it is poor in every script. Layout complexity compounds it."}]},{"t":"p","runs":[{"text":"Measure before you trust it. Take twenty pages representative of your corpus, type the correct text for a few paragraphs by hand, and compute character error rate. Below about 2% you can proceed. Above 10% the retrieval downstream will fail in ways you will misdiagnose as an embedding problem for weeks, because misrecognised words embed to the wrong place and there is no error message anywhere."}]},{"t":"h2","id":"serialise-tables-as-lines-not-as-grids","runs":[{"text":"Serialise tables as lines, not as grids"}]},{"t":"p","runs":[{"text":"A table has to become text. The instinct is to render it as an ASCII grid or a markdown table, and that is usually the wrong choice for a chunked pipeline for one concrete reason: the header row and the data rows get separated when the chunker splits, and a row of numbers with no headings is unrecoverable."}]},{"t":"p","runs":[{"text":"The more robust serialisation repeats the key with every value:"}]},{"t":"code","lang":"","label":"Text","text":"Region: North · Quarter: Q3 2026 · Revenue: 4,120,000 INR · Growth: 3.1%\nRegion: South · Quarter: Q3 2026 · Revenue: 2,880,000 INR · Growth: -0.4%"},{"t":"p","runs":[{"text":"Each line stands alone, survives any split, and embeds meaningfully: a query about southern revenue matches the second line directly. It costs more tokens than a grid, which is a real trade, and it is worth it wherever the table is what people ask about."}]},{"t":"p","runs":[{"text":"For a wide table read as a whole rather than by row, a compact grid inside a single chunk is fine. The rule is: never let a split separate the values from their headings."}]},{"t":"h2","id":"other-formats-briefly","runs":[{"text":"Other formats, briefly"}]},{"t":"p","runs":[{"text":"HTML","b":true},{"text":": strip navigation and boilerplate before anything else, using the statistical method from the previous module. "},{"text":"trafilatura","code":true},{"text":" and "},{"text":"readability-lxml","code":true},{"text":" do main-content extraction well and free."}]},{"t":"p","runs":[{"text":"Office documents","b":true},{"text":": "},{"text":"python-docx","code":true},{"text":" and "},{"text":"openpyxl","code":true},{"text":" read structure directly, which is far better than converting to PDF first. Converting a "},{"text":".docx","code":true},{"text":" to PDF to extract text is throwing away the structure you wanted, then trying to reconstruct it."}]},{"t":"p","runs":[{"text":"Slides","b":true},{"text":": each slide is a chunk, with the deck title and slide number attached. Speaker notes are often the only prose in the file and are frequently forgotten."}]},{"t":"p","runs":[{"text":"Email","b":true},{"text":": quoted history repeats the same text down a thread. Strip quoted blocks or you will index the same paragraph eleven times, with the duplication effects from the previous module."}]},{"t":"h2","id":"store-the-parse-not-just-the-text","runs":[{"text":"Store the parse, not just the text"}]},{"t":"p","runs":[{"text":"Keep, for every document: the extracted text, the extractor and version that produced it, the page or offset each chunk came from, and the character count per page. It is a few extra columns."}]},{"t":"p","runs":[{"text":"What it buys you is the ability to answer, eight months later, the question that is otherwise unanswerable: "},{"text":"did this document parse correctly?","i":true},{"text":" Without it, a corpus of 200,000 documents contains an unknown number of empty and scrambled ones and there is no way to find them except complaints."}]}],"check":{"question":[{"text":"A financial table is extracted into a markdown grid, then chunked at 400 tokens. Answers about specific figures become wrong in a way that looks like hallucination. What is the mechanism?"}],"options":[{"text":[{"text":"The chunker split between the header row and the data rows, leaving numbers with no indication of which column they belong to."}],"correct":true,"tempting":null},{"text":[{"text":"Markdown pipe characters tokenise badly, inflating the token count so fewer rows fit."}],"correct":false,"tempting":[{"text":"Punctuation does cost tokens, but a higher token count would truncate rows visibly rather than misattribute values."}]},{"text":[{"text":"Numbers embed poorly, so the retriever returns the wrong table entirely."}],"correct":false,"tempting":[{"text":"Numeric text does embed weakly, but that would produce answers from an unrelated table rather than plausible wrong values from the right one."}]},{"text":[{"text":"The model cannot perform arithmetic reliably, so derived figures are unreliable regardless of extraction."}],"correct":false,"tempting":[{"text":"A genuine limitation, but the failure described is in reported values rather than in calculations over them."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"saying-what-was-left-out","title":"Admitting the set is partial","minutes":8},"next":{"slug":"chunking-without-losing-meaning","title":"Chunking that does not destroy meaning","minutes":9}}}