Extraction that runs on ten thousand documents

The gap between one document and ten thousand

Extraction demos beautifully. One invoice, one prompt, correct fields, applause. The production version of the same feature has properties the demo does not:

• Some documents are 80 pages and do not fit in one call
• Some are scans of a photocopy of a fax
• Two per cent are not invoices at all
• Somebody needs to know which of the ten thousand results to trust
• It has to finish before Monday and cost less than the person it replaces

Each of those is a design decision, and none of them are about prompting.

Split the document before you split the task

For long documents, the instinct is to send everything and ask for all fields. Better: find the region first, then extract from the region.

A supplier contract’s payment terms live in one clause. Locating that clause (by keyword, by heading, or by an embedding search over sections) and sending 600 tokens instead of 40,000 improves accuracy and cuts cost by a factor of sixty. Recall matters more than precision at the locating stage: send three candidate sections rather than agonising over which one is right.

Where fields live all over the document, extract in groups by locality (everything from the header in one call, line items in another) rather than one call for forty fields. Long field lists degrade towards the end, and a call that produces eight fields well beats one that produces forty with the last ten drifting.

The line-items problem

Repeating rows are the hardest part of document extraction and the least discussed.

Models drop rows, merge rows that wrapped across a page break, and invent a row when a table has a subtotal line. The defences are arithmetic rather than linguistic:

assert len(items) == stated_row_count          # ask for the count separately
assert sum(i.amount_paise for i in items) == total_paise
assert all(i.qty * i.unit_paise == i.amount_paise for i in items)

Ask for the row count as its own field before the rows themselves, so you have two independent statements to compare. Any mismatch routes to review. This catches more real errors than any prompt improvement, and it costs three lines.

Batching, concurrency and cost

Ten thousand documents at one call each is a throughput problem with three levers.

Batch APIs. Most providers offer an asynchronous batch endpoint at roughly half price, with results in up to 24 hours. If the work is not interactive, and extraction rarely is, this halves the bill for no engineering effort beyond writing to a file and polling.

Concurrency with a cap. For interactive or same-day work, a semaphore of 5 to 20 in-flight requests, sized to your rate limit rather than your ambition. Above the limit you generate 429s, retry them, and go slower than if you had been patient.

Cheap model first. Extraction from clean text is exactly the task where a small model performs close to a large one. Route the documents that fail validation to the expensive model. On a typical corpus, 85 per cent never need it.

Always run a pilot of a hundred documents before the full run. Measure accuracy, cost per document and the review rate, then multiply. A team that discovers at document 9,000 that the cost is four times the estimate has burned money it did not have to.

The review queue is the product

No extraction pipeline is right every time, so the real design question is not accuracy: it is what happens to the ones that are wrong.

Sort every result into three lanes:

1. Passed everything: schema valid, arithmetic consistent, quotes verified. Auto-accept.
2. Failed a hard check: totals do not add up, date out of range, required field null. Human review.
3. Passed but flagged: model marked itself low confidence, or the document type was unusual. Sampled review.

Then instrument the queue. Two numbers matter and they are different: review rate (what fraction of documents a person touches) and override rate (what fraction of reviewed documents the person changes). A review rate of 20 per cent with an override rate of 2 per cent means you are wasting reviewer time on documents that were fine: tighten the flagging. An override rate of 40 per cent means your auto-accept lane is probably also wrong and you should widen review, not narrow it.

Make the review screen show the source region beside the extracted value. A reviewer who has to open the original PDF and search for the number reviews at a quarter of the speed, and the throughput of the whole pipeline is set by the human stage, not the model stage.

The corrections are the asset

Every override is a labelled example: the input, the wrong answer, the right answer. Store all three. After a few months you have a few thousand verified pairs, which is:

• A serious evaluation set, drawn from exactly the distribution you serve
• The training data for a distilled small model, if the volume justifies it
• Direct evidence of which failure modes are common, which tells you what to fix next

Most teams throw this away because the reviewer’s correction is written straight into the database with no record of what it replaced. Capturing it costs one extra column and is the most valuable by-product the pipeline produces.