{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"ingestion-pdfs-tables-and-scans","title":"Ingestion: PDFs, tables and scans, where most retrieval actually fails","url":"https://addaly.com/learn/building-with-ai/ingestion-pdfs-tables-and-scans","position":40,"of":80,"minutes":10,"answer":"Most retrieval failures are ingestion failures (interleaved columns, repeated headers, tables flattened into rows that have lost their column names.","keyIdea":"Most retrieval failures are ingestion failures (interleaved columns, repeated headers, tables flattened into rows that have lost their column names, OCR noise and chunks cut from their headings) and the first diagnostic is always to read twenty chunks with your own eyes."},"body":{"format":"markdown","text":"## Where the failure is\n\nTeams spend weeks comparing embedding models while the text in their index is broken. Before touching a model, pull twenty chunks at random from the index and read them. If a chunk starts mid-sentence, contains \"Page 4 of 12 Confidential\" in the middle, or is a run of numbers with no indication of what they measure, no retrieval technique downstream will recover what was lost here. This lesson is about the unglamorous work that decides everything.\n\n## A PDF is not text\n\nA PDF is a set of drawing instructions: put this glyph at this coordinate. There is no paragraph, no reading order, often no space character. A text extractor reconstructs all of that by guessing from positions, and it guesses wrong in predictable ways.\n\nTwo-column layouts come out interleaved, one line from the left column and one from the right, producing sentences that no human wrote. Headers and footers repeat on every page and land inside every chunk, so the string \"Confidential: Internal Use Only\" becomes the most common phrase in your corpus and pollutes every embedding. Words hyphenated across a line break arrive as two fragments. Ligatures such as \"fi\" arrive as a single unusual character that keyword search will never match.\n\nThe free tools are good. `pymupdf` is fast and handles most layouts; `pdfplumber` gives you positions and is the better choice for tables; `pdfminer` is the old reliable. For layout-aware extraction (reading order, headings, tables as tables) `docling` from IBM and `marker` are free and markedly better than plain extraction on real documents. Strip repeated headers by finding lines that occur on more than half the pages at the same position, and drop them.\n\n## Tables lose their heads\n\nA table flattened row by row is the single most damaging thing a naive pipeline does. Row forty of a pricing table becomes the chunk \"Pro 50 100 Yes 24h\", and neither an embedding nor a reader knows that 50 is users, 100 is projects and 24h is support response. The column headers were in the chunk three hundred rows earlier.\n\nConvert tables to a form that carries the headers with every row. Markdown tables work for short ones. For long ones, emit one line per row in \"header: value\" form:\n\n```text\nPlan: Pro. Users: 50. Projects: 100. SSO: Yes. Support response: 24h.\n```\n\nFor large tables of real data (a price list with thousands of rows) do not embed them at all. Load them into a database and give the model a tool that queries it, which the tool-use module already taught you how to build. A model reading a table row by row is the wrong instrument for a lookup.\n\n## Scans and OCR\n\nA scanned document has no text layer, and the only route in is optical character recognition. Tesseract is free and adequate on clean, printed English; it ships trained data for Hindi, Tamil, Bengali and most Indian scripts, with noticeably lower accuracy than on Latin text. PaddleOCR is a stronger free option on complex layouts. Hosted document services from Google, Microsoft and Mistral cost in the region of a dollar or two per thousand pages and are better on poor scans, handwriting and mixed scripts.\n\nOCR errors are not cosmetic. A keyword search fails completely on \"invoicc\", and an embedding of a paragraph with a five per cent character error rate drifts measurably. Keep the OCR confidence score per page, flag pages below a threshold for human review, and store a reference to the page image so an answer can show the original rather than the transcription.\n\n## The heading a chunk forgot\n\nA chunk that reads \"The limit is 500 per month\" is useless without knowing it sits under \"Enterprise plan\" in \"API rate limits\". Chunking cut it from its heading path. Prepend the path (document title, then each heading level down to the chunk) to the chunk text before embedding. It costs twenty tokens per chunk and reliably improves recall, because the chunk now says what it is about.\n\n> Figure: Twenty chunks read by eye, before and after. What the first pipeline stored: Pro 50 100 Yes 24h; Confidential: Internal Use Only; This does not apply to part-time staff; Page 4 of 12; The limit is 500 per month. What the same content should say: Plan: Pro. Users: 50. Projects: 100. SSO: Yes. Support response: 24h.; Dropped: a line appearing on more than half the pages at the same position; Leave policy > Part-time staff: This does not apply to part-time staff; Dropped with the running header; API rate limits > Enterprise plan: The limit is 500 per month. None of the fixes on the right needed a different embedding model. Read twenty chunks at random after every parser change: it takes ten minutes and it catches what no metric does.\n\nA stronger version has a model write one sentence of context per chunk, describing where it sits in the document, and prepends that instead. Anthropic reported a large reduction in failed retrievals from this combined with keyword search; the cost is one model call per chunk at ingestion, which on a million chunks is real money and real time, so measure the gain on your own set before committing to it.\n\n## Other formats, briefly\n\nHTML needs its navigation, cookie banners and footers removed; `trafilatura` and readability-style extractors do this. Word documents go through `python-docx` or `pandoc`. Spreadsheets are records, not prose; treat them like the large-table case. Code should be chunked on function and class boundaries, which `tree-sitter` finds for any language, not on line counts.\n\n## Duplicates and provenance\n\nNear-duplicates (the same policy pasted into four documents) survive into the index and crowd out variety in results. MinHash over shingles at ingestion catches them cheaply.\n\nEvery chunk must carry its provenance: source document id, page, character offset, heading path, ingestion time and parser version. That is what lets an answer cite a page, lets you re-parse only the documents a parser bug affected, and lets you re-embed when the model changes. Provenance you did not store at ingestion cannot be reconstructed later.\n\n## The eyeball test, again\n\nAfter every parser change, sample twenty chunks and read them. It takes ten minutes and it catches what no metric does."},"check":{"question":"A support bot over a pricing PDF cannot answer \"how many projects does the Pro plan allow?\" even though the retrieved chunk contains the row for Pro. What has happened?","options":[{"text":"The table was flattened row by row, so the chunk holds the values but not the column headers that give them meaning.","correct":true},{"text":"The embedding model does not handle numbers well, so the row was retrieved by chance and the model cannot interpret digits.","correct":false,"tempting":"The row was retrieved correctly; the failure is in what the chunk contains, not in how numbers embed."},{"text":"The chunk is too short and needs to be merged with the surrounding rows to give the model enough context.","correct":false,"tempting":"Neighbouring rows carry more values with the same missing headers; the missing information is hundreds of rows away."},{"text":"The PDF's two-column layout has interleaved the pricing table with the text beside it.","correct":false,"tempting":"A real PDF failure, but it produces garbled sentences rather than a clean row of unlabelled values."}]},"prev":{"slug":"rewriting-the-question","title":"Rewriting the question before you search it","url":"https://addaly.com/learn/building-with-ai/rewriting-the-question"},"next":{"slug":"permissions-freshness-and-metadata","title":"Permissions, freshness and metadata: the boring half that leaks data","url":"https://addaly.com/learn/building-with-ai/permissions-freshness-and-metadata"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ingestion: PDFs, tables and scans, where most retrieval actually fails', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/ingestion-pdfs-tables-and-scans (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ingestion: PDFs, tables and scans, where most retrieval actually fails","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"Most retrieval failures are ingestion failures (interleaved columns, repeated headers, tables flattened into rows that have lost their column names, OCR noise and chunks cut from their headings) and the first diagnostic is always to read twenty chunks with your own eyes.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"What the first pipeline stored","items":["Pro 50 100 Yes 24h","Confidential: Internal Use Only","This does not apply to part-time staff","Page 4 of 12","The limit is 500 per month"]},"type":"compare","right":{"title":"What the same content should say","items":["Plan: Pro. Users: 50. Projects: 100. SSO: Yes. Support response: 24h.","Dropped: a line appearing on more than half the pages at the same position","Leave policy > Part-time staff: This does not apply to part-time staff","Dropped with the running header","API rate limits > Enterprise plan: The limit is 500 per month"]},"title":"Twenty chunks read by eye, before and after","caption":"None of the fixes on the right needed a different embedding model. Read twenty chunks at random after every parser change: it takes ten minutes and it catches what no metric does."},"alt":"Figure: Twenty chunks read by eye, before and after. What the first pipeline stored: Pro 50 100 Yes 24h; Confidential: Internal Use Only; This does not apply to part-time staff; Page 4 of 12; The limit is 500 per month. What the same content should say: Plan: Pro. Users: 50. Projects: 100. SSO: Yes. Support response: 24h.; Dropped: a line appearing on more than half the pages at the same position; Leave policy > Part-time staff: This does not apply to part-time staff; Dropped with the running header; API rate limits > Enterprise plan: The limit is 500 per month. None of the fixes on the right needed a different embedding model. Read twenty chunks at random after every parser change: it takes ten minutes and it catches what no metric does."}},"blocks":[{"t":"h2","id":"where-the-failure-is","runs":[{"text":"Where the failure is"}]},{"t":"p","runs":[{"text":"Teams spend weeks comparing embedding models while the text in their index is broken. Before touching a model, pull twenty chunks at random from the index and read them. If a chunk starts mid-sentence, contains “Page 4 of 12 Confidential” in the middle, or is a run of numbers with no indication of what they measure, no retrieval technique downstream will recover what was lost here. This lesson is about the unglamorous work that decides everything."}]},{"t":"h2","id":"a-pdf-is-not-text","runs":[{"text":"A PDF is not text"}]},{"t":"p","runs":[{"text":"A PDF is a set of drawing instructions: put this glyph at this coordinate. There is no paragraph, no reading order, often no space character. A text extractor reconstructs all of that by guessing from positions, and it guesses wrong in predictable ways."}]},{"t":"p","runs":[{"text":"Two-column layouts come out interleaved, one line from the left column and one from the right, producing sentences that no human wrote. Headers and footers repeat on every page and land inside every chunk, so the string “Confidential: Internal Use Only” becomes the most common phrase in your corpus and pollutes every embedding. Words hyphenated across a line break arrive as two fragments. Ligatures such as “fi” arrive as a single unusual character that keyword search will never match."}]},{"t":"p","runs":[{"text":"The free tools are good. "},{"text":"pymupdf","code":true},{"text":" is fast and handles most layouts; "},{"text":"pdfplumber","code":true},{"text":" gives you positions and is the better choice for tables; "},{"text":"pdfminer","code":true},{"text":" is the old reliable. For layout-aware extraction (reading order, headings, tables as tables) "},{"text":"docling","code":true},{"text":" from IBM and "},{"text":"marker","code":true},{"text":" are free and markedly better than plain extraction on real documents. Strip repeated headers by finding lines that occur on more than half the pages at the same position, and drop them."}]},{"t":"h2","id":"tables-lose-their-heads","runs":[{"text":"Tables lose their heads"}]},{"t":"p","runs":[{"text":"A table flattened row by row is the single most damaging thing a naive pipeline does. Row forty of a pricing table becomes the chunk “Pro 50 100 Yes 24h”, and neither an embedding nor a reader knows that 50 is users, 100 is projects and 24h is support response. The column headers were in the chunk three hundred rows earlier."}]},{"t":"p","runs":[{"text":"Convert tables to a form that carries the headers with every row. Markdown tables work for short ones. For long ones, emit one line per row in “header: value” form:"}]},{"t":"code","lang":"text","label":"Text","text":"Plan: Pro. Users: 50. Projects: 100. SSO: Yes. Support response: 24h."},{"t":"p","runs":[{"text":"For large tables of real data (a price list with thousands of rows) do not embed them at all. Load them into a database and give the model a tool that queries it, which the tool-use module already taught you how to build. A model reading a table row by row is the wrong instrument for a lookup."}]},{"t":"h2","id":"scans-and-ocr","runs":[{"text":"Scans and OCR"}]},{"t":"p","runs":[{"text":"A scanned document has no text layer, and the only route in is optical character recognition. Tesseract is free and adequate on clean, printed English; it ships trained data for Hindi, Tamil, Bengali and most Indian scripts, with noticeably lower accuracy than on Latin text. PaddleOCR is a stronger free option on complex layouts. Hosted document services from Google, Microsoft and Mistral cost in the region of a dollar or two per thousand pages and are better on poor scans, handwriting and mixed scripts."}]},{"t":"p","runs":[{"text":"OCR errors are not cosmetic. A keyword search fails completely on “invoicc”, and an embedding of a paragraph with a five per cent character error rate drifts measurably. Keep the OCR confidence score per page, flag pages below a threshold for human review, and store a reference to the page image so an answer can show the original rather than the transcription."}]},{"t":"h2","id":"the-heading-a-chunk-forgot","runs":[{"text":"The heading a chunk forgot"}]},{"t":"p","runs":[{"text":"A chunk that reads “The limit is 500 per month” is useless without knowing it sits under “Enterprise plan” in “API rate limits”. Chunking cut it from its heading path. Prepend the path (document title, then each heading level down to the chunk) to the chunk text before embedding. It costs twenty tokens per chunk and reliably improves recall, because the chunk now says what it is about."}]},{"t":"p","runs":[{"text":"A stronger version has a model write one sentence of context per chunk, describing where it sits in the document, and prepends that instead. Anthropic reported a large reduction in failed retrievals from this combined with keyword search; the cost is one model call per chunk at ingestion, which on a million chunks is real money and real time, so measure the gain on your own set before committing to it."}]},{"t":"h2","id":"other-formats-briefly","runs":[{"text":"Other formats, briefly"}]},{"t":"p","runs":[{"text":"HTML needs its navigation, cookie banners and footers removed; "},{"text":"trafilatura","code":true},{"text":" and readability-style extractors do this. Word documents go through "},{"text":"python-docx","code":true},{"text":" or "},{"text":"pandoc","code":true},{"text":". Spreadsheets are records, not prose; treat them like the large-table case. Code should be chunked on function and class boundaries, which "},{"text":"tree-sitter","code":true},{"text":" finds for any language, not on line counts."}]},{"t":"h2","id":"duplicates-and-provenance","runs":[{"text":"Duplicates and provenance"}]},{"t":"p","runs":[{"text":"Near-duplicates (the same policy pasted into four documents) survive into the index and crowd out variety in results. MinHash over shingles at ingestion catches them cheaply."}]},{"t":"p","runs":[{"text":"Every chunk must carry its provenance: source document id, page, character offset, heading path, ingestion time and parser version. That is what lets an answer cite a page, lets you re-parse only the documents a parser bug affected, and lets you re-embed when the model changes. Provenance you did not store at ingestion cannot be reconstructed later."}]},{"t":"h2","id":"the-eyeball-test-again","runs":[{"text":"The eyeball test, again"}]},{"t":"p","runs":[{"text":"After every parser change, sample twenty chunks and read them. It takes ten minutes and it catches what no metric does."}]}],"check":{"question":[{"text":"A support bot over a pricing PDF cannot answer “how many projects does the Pro plan allow?” even though the retrieved chunk contains the row for Pro. What has happened?"}],"options":[{"text":[{"text":"The table was flattened row by row, so the chunk holds the values but not the column headers that give them meaning."}],"correct":true,"tempting":null},{"text":[{"text":"The embedding model does not handle numbers well, so the row was retrieved by chance and the model cannot interpret digits."}],"correct":false,"tempting":[{"text":"The row was retrieved correctly; the failure is in what the chunk contains, not in how numbers embed."}]},{"text":[{"text":"The chunk is too short and needs to be merged with the surrounding rows to give the model enough context."}],"correct":false,"tempting":[{"text":"Neighbouring rows carry more values with the same missing headers; the missing information is hundreds of rows away."}]},{"text":[{"text":"The PDF’s two-column layout has interleaved the pricing table with the text beside it."}],"correct":false,"tempting":[{"text":"A real PDF failure, but it produces garbled sentences rather than a clean row of unlabelled values."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"rewriting-the-question","title":"Rewriting the question before you search it","minutes":9},"next":{"slug":"permissions-freshness-and-metadata","title":"Permissions, freshness and metadata: the boring half that leaks data","minutes":9}}}