Documents: stages that multiply, and checks that need no labels

Three stages, and the multiplication

A document pipeline is usually: get the pixels right, get the structure right, get the fields right. Each stage looks respectable and the product of three respectable stages is not.

0.97 (page classification) * 0.95 (text and layout) * 0.94 (field extraction) = 0.87

Thirteen documents in a hundred go wrong, and the end-to-end number is the only one the customer experiences. Measure every stage, because that is where the fix lives, and always report the end-to-end figure alongside, because that is the promise.

Character accuracy is not reading accuracy

An OCR engine can transcribe every character on a page correctly and produce useless output, because it read a two-column page straight across, or emitted a table row by row when the meaning was column by column. Character error rate cannot see this: the characters are all present.

So measure reading order and structure separately from characters:

• Reading order: does the extracted text sequence match the human reading sequence, checked on a sample by a person.
• Table structure: start with the simplest counts, row count and column count correct, because those errors explain almost every downstream failure. Then cell-level precision and recall, matched by position, with merged and spanning cells recorded as their own error category.

Build the set from real intake

This is the single largest source of self-deception in document AI. An evaluation assembled from clean, digitally-generated PDFs measures a system that will meet photographs of paper.

Real intake looks like: a phone photo at fifteen degrees, glare across the total, a staple through the date, a rubber stamp over the signature, handwriting in the margin, a faded thermal print, a fax at 200 dpi, page three photographed twice and page four missing.

Take a hundred documents from the actual intake channel, keep the ugly ones, and note the condition of each so you can slice by it. The distribution of conditions is itself a finding, and it usually argues for fixing the capture screen before touching the model.

Checks the document performs on itself

Documents carry internal redundancy, and it gives you correctness checks with no gold labels at all, which means they run on live traffic, not only on your set.

• Line items sum to the subtotal; subtotal plus tax equals the total.
• The tax rate applied matches the stated rate.
• Dates are ordered sensibly: issue before due, dispatch before delivery.
• Identifiers pass their checksum, where the format has one.
• The page count matches the “page 1 of 4” in the header.

An arithmetic failure means something was misread even though you have no idea what the truth is. Log the failure rate of these checks as a production quality metric; it moves the moment an upstream capture setting or a supplier’s template changes, and it costs nothing per document.

Templates are the hidden cluster

Document sets are dominated by a handful of layouts. Two hundred invoices from one large supplier share a template, so a model that has learned that template scores well on forty per cent of your set while being useless on everything else. Group the set by supplier or template, report accuracy per template group, and count how many distinct templates the set actually contains. A set of 500 documents from 12 templates has twelve independent observations of layout handling, which is the number that governs how well a new supplier will go.

Multi-page reality

Batch-scanned intake arrives as a stack, and the pipeline has to decide where one document ends and the next begins. The failure mode is not a wrong field but a wrong boundary: two invoices merged into one record, or a three-page contract split into two.

Measure document-splitting accuracy explicitly (correct boundaries found, spurious boundaries added) before field-level metrics, because a split error makes every field metric downstream meaningless.

Scripts, and where the tooling is thin

OCR quality for Devanagari, Bengali, Tamil and other Indic scripts lags Latin substantially, and mixed-script documents (an English form filled in Hindi) are harder still. Report per script, and expect the gap.

Free tools take you a long way: Tesseract as a baseline that runs anywhere, PaddleOCR and docTR for better accuracy including several non-Latin scripts, and jiwer for character error rate. Run the free baseline even if you intend to buy, because the difference between it and a paid service on your documents is the only version of that comparison worth having.