{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-documents-and-ocr","title":"Documents: stages that multiply, and checks that need no labels","url":"https://addaly.com/learn/evaluating-ai/evaluating-documents-and-ocr","position":38,"of":76,"minutes":9,"answer":"A document pipeline's accuracy is the product of its stages, so measure each one and the end-to-end separately, and use the document's own internal consistency, such as line items summing to the total.","keyIdea":"A document pipeline's accuracy is the product of its stages, so measure each one and the end-to-end separately, and use the document's own internal consistency, such as line items summing to the total, as a label-free check on live traffic."},"body":{"format":"markdown","text":"## Three stages, and the multiplication\n\nA document pipeline is usually: get the pixels right, get the structure right, get the fields right. Each stage looks respectable and the product of three respectable stages is not.\n\n```\n0.97 (page classification) * 0.95 (text and layout) * 0.94 (field extraction) = 0.87\n```\n\nThirteen documents in a hundred go wrong, and the end-to-end number is the only one the customer experiences. Measure every stage, because that is where the fix lives, and always report the end-to-end figure alongside, because that is the promise.\n\n> Figure: Three respectable stages, one product. Bars (% correct): Page classification 97; Text and layout 95; Field extraction 94; End to end: the product 87. 0.97 × 0.95 × 0.94 = 0.87. Thirteen documents in a hundred go wrong, and the customer only ever meets the last bar.\n\n## Character accuracy is not reading accuracy\n\nAn OCR engine can transcribe every character on a page correctly and produce useless output, because it read a two-column page straight across, or emitted a table row by row when the meaning was column by column. Character error rate cannot see this: the characters are all present.\n\nSo measure reading order and structure separately from characters:\n\n- **Reading order**: does the extracted text sequence match the human reading sequence, checked on a sample by a person.\n- **Table structure**: start with the simplest counts, row count and column count correct, because those errors explain almost every downstream failure. Then cell-level precision and recall, matched by position, with merged and spanning cells recorded as their own error category.\n\n## Build the set from real intake\n\nThis is the single largest source of self-deception in document AI. An evaluation assembled from clean, digitally-generated PDFs measures a system that will meet photographs of paper.\n\nReal intake looks like: a phone photo at fifteen degrees, glare across the total, a staple through the date, a rubber stamp over the signature, handwriting in the margin, a faded thermal print, a fax at 200 dpi, page three photographed twice and page four missing.\n\nTake a hundred documents from the actual intake channel, keep the ugly ones, and note the condition of each so you can slice by it. The distribution of conditions is itself a finding, and it usually argues for fixing the capture screen before touching the model.\n\n## Checks the document performs on itself\n\nDocuments carry internal redundancy, and it gives you correctness checks with no gold labels at all, which means they run on live traffic, not only on your set.\n\n- Line items sum to the subtotal; subtotal plus tax equals the total.\n- The tax rate applied matches the stated rate.\n- Dates are ordered sensibly: issue before due, dispatch before delivery.\n- Identifiers pass their checksum, where the format has one.\n- The page count matches the \"page 1 of 4\" in the header.\n\nAn arithmetic failure means something was misread even though you have no idea what the truth is. Log the failure rate of these checks as a production quality metric; it moves the moment an upstream capture setting or a supplier's template changes, and it costs nothing per document.\n\n## Templates are the hidden cluster\n\nDocument sets are dominated by a handful of layouts. Two hundred invoices from one large supplier share a template, so a model that has learned that template scores well on forty per cent of your set while being useless on everything else. Group the set by supplier or template, report accuracy per template group, and count how many distinct templates the set actually contains. A set of 500 documents from 12 templates has twelve independent observations of layout handling, which is the number that governs how well a new supplier will go.\n\n## Multi-page reality\n\nBatch-scanned intake arrives as a stack, and the pipeline has to decide where one document ends and the next begins. The failure mode is not a wrong field but a wrong boundary: two invoices merged into one record, or a three-page contract split into two.\n\nMeasure document-splitting accuracy explicitly (correct boundaries found, spurious boundaries added) before field-level metrics, because a split error makes every field metric downstream meaningless.\n\n## Scripts, and where the tooling is thin\n\nOCR quality for Devanagari, Bengali, Tamil and other Indic scripts lags Latin substantially, and mixed-script documents (an English form filled in Hindi) are harder still. Report per script, and expect the gap.\n\nFree tools take you a long way: Tesseract as a baseline that runs anywhere, PaddleOCR and docTR for better accuracy including several non-Latin scripts, and `jiwer` for character error rate. Run the free baseline even if you intend to buy, because the difference between it and a paid service on *your* documents is the only version of that comparison worth having."},"check":{"question":"Why is an internal consistency check, such as line items summing to the invoice total, unusually valuable in a document pipeline?","options":[{"text":"It measures extraction quality more accurately than field-level comparison against gold labels.","correct":false,"tempting":"Claims more than the check can deliver: it detects a class of errors and stays silent on fields with no arithmetic relationship."},{"text":"It requires no gold labels, so it can run on live production documents and flag misreads for which no ground truth exists.","correct":true},{"text":"It replaces the need for a labelled evaluation set once the pipeline is in production.","correct":false,"tempting":"Treats a monitoring signal as a substitute for measurement; consistency checks cannot tell you the supplier name is wrong."},{"text":"It detects OCR errors specifically, allowing the OCR stage to be scored separately from the extraction stage.","correct":false,"tempting":"An arithmetic mismatch does not identify which stage failed, since a layout or extraction error produces the same symptom."}]},"prev":{"slug":"evaluating-speech","title":"Speech: word error rate, and the words that matter","url":"https://addaly.com/learn/evaluating-ai/evaluating-speech"},"next":{"slug":"evaluating-tabular-models","title":"Tables, cross-validation, and the leak that gives you 0.99","url":"https://addaly.com/learn/evaluating-ai/evaluating-tabular-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Documents: stages that multiply, and checks that need no labels', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-documents-and-ocr (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Documents: stages that multiply, and checks that need no labels","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"A document pipeline’s accuracy is the product of its stages, so measure each one and the end-to-end separately, and use the document’s own internal consistency, such as line items summing to the total, as a label-free check on live traffic.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% correct","title":"Three respectable stages, one product","series":[{"label":"Page classification","value":97},{"label":"Text and layout","value":95},{"label":"Field extraction","value":94},{"label":"End to end: the product","value":87,"highlight":true}],"caption":"0.97 × 0.95 × 0.94 = 0.87. Thirteen documents in a hundred go wrong, and the customer only ever meets the last bar."},"alt":"Figure: Three respectable stages, one product. Bars (% correct): Page classification 97; Text and layout 95; Field extraction 94; End to end: the product 87. 0.97 × 0.95 × 0.94 = 0.87. Thirteen documents in a hundred go wrong, and the customer only ever meets the last bar."}},"blocks":[{"t":"h2","id":"three-stages-and-the-multiplication","runs":[{"text":"Three stages, and the multiplication"}]},{"t":"p","runs":[{"text":"A document pipeline is usually: get the pixels right, get the structure right, get the fields right. Each stage looks respectable and the product of three respectable stages is not."}]},{"t":"code","lang":"","label":"Text","text":"0.97 (page classification) * 0.95 (text and layout) * 0.94 (field extraction) = 0.87"},{"t":"p","runs":[{"text":"Thirteen documents in a hundred go wrong, and the end-to-end number is the only one the customer experiences. Measure every stage, because that is where the fix lives, and always report the end-to-end figure alongside, because that is the promise."}]},{"t":"h2","id":"character-accuracy-is-not-reading-accuracy","runs":[{"text":"Character accuracy is not reading accuracy"}]},{"t":"p","runs":[{"text":"An OCR engine can transcribe every character on a page correctly and produce useless output, because it read a two-column page straight across, or emitted a table row by row when the meaning was column by column. Character error rate cannot see this: the characters are all present."}]},{"t":"p","runs":[{"text":"So measure reading order and structure separately from characters:"}]},{"t":"ul","items":[[{"text":"Reading order","b":true},{"text":": does the extracted text sequence match the human reading sequence, checked on a sample by a person."}],[{"text":"Table structure","b":true},{"text":": start with the simplest counts, row count and column count correct, because those errors explain almost every downstream failure. Then cell-level precision and recall, matched by position, with merged and spanning cells recorded as their own error category."}]]},{"t":"h2","id":"build-the-set-from-real-intake","runs":[{"text":"Build the set from real intake"}]},{"t":"p","runs":[{"text":"This is the single largest source of self-deception in document AI. An evaluation assembled from clean, digitally-generated PDFs measures a system that will meet photographs of paper."}]},{"t":"p","runs":[{"text":"Real intake looks like: a phone photo at fifteen degrees, glare across the total, a staple through the date, a rubber stamp over the signature, handwriting in the margin, a faded thermal print, a fax at 200 dpi, page three photographed twice and page four missing."}]},{"t":"p","runs":[{"text":"Take a hundred documents from the actual intake channel, keep the ugly ones, and note the condition of each so you can slice by it. The distribution of conditions is itself a finding, and it usually argues for fixing the capture screen before touching the model."}]},{"t":"h2","id":"checks-the-document-performs-on-itself","runs":[{"text":"Checks the document performs on itself"}]},{"t":"p","runs":[{"text":"Documents carry internal redundancy, and it gives you correctness checks with no gold labels at all, which means they run on live traffic, not only on your set."}]},{"t":"ul","items":[[{"text":"Line items sum to the subtotal; subtotal plus tax equals the total."}],[{"text":"The tax rate applied matches the stated rate."}],[{"text":"Dates are ordered sensibly: issue before due, dispatch before delivery."}],[{"text":"Identifiers pass their checksum, where the format has one."}],[{"text":"The page count matches the “page 1 of 4” in the header."}]]},{"t":"p","runs":[{"text":"An arithmetic failure means something was misread even though you have no idea what the truth is. Log the failure rate of these checks as a production quality metric; it moves the moment an upstream capture setting or a supplier’s template changes, and it costs nothing per document."}]},{"t":"h2","id":"templates-are-the-hidden-cluster","runs":[{"text":"Templates are the hidden cluster"}]},{"t":"p","runs":[{"text":"Document sets are dominated by a handful of layouts. Two hundred invoices from one large supplier share a template, so a model that has learned that template scores well on forty per cent of your set while being useless on everything else. Group the set by supplier or template, report accuracy per template group, and count how many distinct templates the set actually contains. A set of 500 documents from 12 templates has twelve independent observations of layout handling, which is the number that governs how well a new supplier will go."}]},{"t":"h2","id":"multi-page-reality","runs":[{"text":"Multi-page reality"}]},{"t":"p","runs":[{"text":"Batch-scanned intake arrives as a stack, and the pipeline has to decide where one document ends and the next begins. The failure mode is not a wrong field but a wrong boundary: two invoices merged into one record, or a three-page contract split into two."}]},{"t":"p","runs":[{"text":"Measure document-splitting accuracy explicitly (correct boundaries found, spurious boundaries added) before field-level metrics, because a split error makes every field metric downstream meaningless."}]},{"t":"h2","id":"scripts-and-where-the-tooling-is-thin","runs":[{"text":"Scripts, and where the tooling is thin"}]},{"t":"p","runs":[{"text":"OCR quality for Devanagari, Bengali, Tamil and other Indic scripts lags Latin substantially, and mixed-script documents (an English form filled in Hindi) are harder still. Report per script, and expect the gap."}]},{"t":"p","runs":[{"text":"Free tools take you a long way: Tesseract as a baseline that runs anywhere, PaddleOCR and docTR for better accuracy including several non-Latin scripts, and "},{"text":"jiwer","code":true},{"text":" for character error rate. Run the free baseline even if you intend to buy, because the difference between it and a paid service on "},{"text":"your","i":true},{"text":" documents is the only version of that comparison worth having."}]}],"check":{"question":[{"text":"Why is an internal consistency check, such as line items summing to the invoice total, unusually valuable in a document pipeline?"}],"options":[{"text":[{"text":"It measures extraction quality more accurately than field-level comparison against gold labels."}],"correct":false,"tempting":[{"text":"Claims more than the check can deliver: it detects a class of errors and stays silent on fields with no arithmetic relationship."}]},{"text":[{"text":"It requires no gold labels, so it can run on live production documents and flag misreads for which no ground truth exists."}],"correct":true,"tempting":null},{"text":[{"text":"It replaces the need for a labelled evaluation set once the pipeline is in production."}],"correct":false,"tempting":[{"text":"Treats a monitoring signal as a substitute for measurement; consistency checks cannot tell you the supplier name is wrong."}]},{"text":[{"text":"It detects OCR errors specifically, allowing the OCR stage to be scored separately from the extraction stage."}],"correct":false,"tempting":[{"text":"An arithmetic mismatch does not identify which stage failed, since a layout or extraction error produces the same symptom."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"evaluating-speech","title":"Speech: word error rate, and the words that matter","minutes":9},"next":{"slug":"evaluating-tabular-models","title":"Tables, cross-validation, and the leak that gives you 0.99","minutes":9}}}