{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"extraction-at-scale","title":"Extraction that runs on ten thousand documents","url":"https://addaly.com/learn/building-with-ai/extraction-at-scale","position":23,"of":80,"minutes":9,"answer":"Locate the region before extracting from it, verify repeating rows with arithmetic rather than prompting, and design the review queue and its override log as the actual product.","keyIdea":"Locate the region before extracting from it, verify repeating rows with arithmetic rather than prompting, and design the review queue and its override log as the actual product, because the corrections are your evaluation set and your training data."},"body":{"format":"markdown","text":"## The gap between one document and ten thousand\n\nExtraction demos beautifully. One invoice, one prompt, correct fields, applause. The production version of the same feature has properties the demo does not:\n\n- Some documents are 80 pages and do not fit in one call\n- Some are scans of a photocopy of a fax\n- Two per cent are not invoices at all\n- Somebody needs to know which of the ten thousand results to trust\n- It has to finish before Monday and cost less than the person it replaces\n\nEach of those is a design decision, and none of them are about prompting.\n\n## Split the document before you split the task\n\nFor long documents, the instinct is to send everything and ask for all fields. Better: find the region first, then extract from the region.\n\nA supplier contract's payment terms live in one clause. Locating that clause (by keyword, by heading, or by an embedding search over sections) and sending 600 tokens instead of 40,000 improves accuracy and cuts cost by a factor of sixty. Recall matters more than precision at the locating stage: send three candidate sections rather than agonising over which one is right.\n\nWhere fields live all over the document, extract in **groups by locality** (everything from the header in one call, line items in another) rather than one call for forty fields. Long field lists degrade towards the end, and a call that produces eight fields well beats one that produces forty with the last ten drifting.\n\n## The line-items problem\n\nRepeating rows are the hardest part of document extraction and the least discussed.\n\nModels drop rows, merge rows that wrapped across a page break, and invent a row when a table has a subtotal line. The defences are arithmetic rather than linguistic:\n\n```python\nassert len(items) == stated_row_count          # ask for the count separately\nassert sum(i.amount_paise for i in items) == total_paise\nassert all(i.qty * i.unit_paise == i.amount_paise for i in items)\n```\n\nAsk for the row count as its own field before the rows themselves, so you have two independent statements to compare. Any mismatch routes to review. This catches more real errors than any prompt improvement, and it costs three lines.\n\n## Batching, concurrency and cost\n\nTen thousand documents at one call each is a throughput problem with three levers.\n\n**Batch APIs.** Most providers offer an asynchronous batch endpoint at roughly half price, with results in up to 24 hours. If the work is not interactive, and extraction rarely is, this halves the bill for no engineering effort beyond writing to a file and polling.\n\n**Concurrency with a cap.** For interactive or same-day work, a semaphore of 5 to 20 in-flight requests, sized to your rate limit rather than your ambition. Above the limit you generate 429s, retry them, and go slower than if you had been patient.\n\n**Cheap model first.** Extraction from clean text is exactly the task where a small model performs close to a large one. Route the documents that fail validation to the expensive model. On a typical corpus, 85 per cent never need it.\n\nAlways run a **pilot of a hundred documents** before the full run. Measure accuracy, cost per document and the review rate, then multiply. A team that discovers at document 9,000 that the cost is four times the estimate has burned money it did not have to.\n\n## The review queue is the product\n\nNo extraction pipeline is right every time, so the real design question is not accuracy: it is what happens to the ones that are wrong.\n\nSort every result into three lanes:\n\n1. **Passed everything**: schema valid, arithmetic consistent, quotes verified. Auto-accept.\n2. **Failed a hard check**: totals do not add up, date out of range, required field null. Human review.\n3. **Passed but flagged**: model marked itself low confidence, or the document type was unusual. Sampled review.\n\nThen instrument the queue. Two numbers matter and they are different: **review rate** (what fraction of documents a person touches) and **override rate** (what fraction of reviewed documents the person changes). A review rate of 20 per cent with an override rate of 2 per cent means you are wasting reviewer time on documents that were fine: tighten the flagging. An override rate of 40 per cent means your auto-accept lane is probably also wrong and you should widen review, not narrow it.\n\nMake the review screen show the source region beside the extracted value. A reviewer who has to open the original PDF and search for the number reviews at a quarter of the speed, and the throughput of the whole pipeline is set by the human stage, not the model stage.\n\n## The corrections are the asset\n\nEvery override is a labelled example: the input, the wrong answer, the right answer. Store all three. After a few months you have a few thousand verified pairs, which is:\n\n- A serious evaluation set, drawn from exactly the distribution you serve\n- The training data for a distilled small model, if the volume justifies it\n- Direct evidence of which failure modes are common, which tells you what to fix next\n\nMost teams throw this away because the reviewer's correction is written straight into the database with no record of what it replaced. Capturing it costs one extra column and is the most valuable by-product the pipeline produces."},"check":{"question":"An invoice pipeline reports 97% field accuracy in testing but finance finds recurring errors in line items. Which change addresses the mechanism rather than the symptom?","options":[{"text":"Raise the model tier so line items are read more carefully.","correct":false,"tempting":"Improves average reading without adding any way to detect a dropped or merged row, which is the specific failure with repeating structures."},{"text":"Send each page separately so tables are never split across calls.","correct":false,"tempting":"Helps with page-break merges but creates a new problem, since a table split across pages now has no call that sees the whole of it."},{"text":"Ask for the row count as its own field and assert that the items sum to the stated total, routing any mismatch to review.","correct":true},{"text":"Increase the field-level accuracy target from 97% to 99% before shipping.","correct":false,"tempting":"Sets a stricter goal without supplying a mechanism to reach it or to catch the residual errors that will still occur."}]},"prev":{"slug":"schemas-models-can-follow","title":"Designing a schema a model can actually fill","url":"https://addaly.com/learn/building-with-ai/schemas-models-can-follow"},"next":{"slug":"tool-use","title":"Tool use: the model never runs anything","url":"https://addaly.com/learn/building-with-ai/tool-use"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Extraction that runs on ten thousand documents', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/extraction-at-scale (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Extraction that runs on ten thousand documents","courseTitle":"Building with AI","module":{"index":3,"slug":"structure-tools-and-the-loop","title":"Structure, tools and the loop"},"answer":null,"keyIdea":"Locate the region before extracting from it, verify repeating rows with arithmetic rather than prompting, and design the review queue and its override log as the actual product, because the corrections are your evaluation set and your training data.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-gap-between-one-document-and-ten-thousand","runs":[{"text":"The gap between one document and ten thousand"}]},{"t":"p","runs":[{"text":"Extraction demos beautifully. One invoice, one prompt, correct fields, applause. The production version of the same feature has properties the demo does not:"}]},{"t":"ul","items":[[{"text":"Some documents are 80 pages and do not fit in one call"}],[{"text":"Some are scans of a photocopy of a fax"}],[{"text":"Two per cent are not invoices at all"}],[{"text":"Somebody needs to know which of the ten thousand results to trust"}],[{"text":"It has to finish before Monday and cost less than the person it replaces"}]]},{"t":"p","runs":[{"text":"Each of those is a design decision, and none of them are about prompting."}]},{"t":"h2","id":"split-the-document-before-you-split-the-task","runs":[{"text":"Split the document before you split the task"}]},{"t":"p","runs":[{"text":"For long documents, the instinct is to send everything and ask for all fields. Better: find the region first, then extract from the region."}]},{"t":"p","runs":[{"text":"A supplier contract’s payment terms live in one clause. Locating that clause (by keyword, by heading, or by an embedding search over sections) and sending 600 tokens instead of 40,000 improves accuracy and cuts cost by a factor of sixty. Recall matters more than precision at the locating stage: send three candidate sections rather than agonising over which one is right."}]},{"t":"p","runs":[{"text":"Where fields live all over the document, extract in "},{"text":"groups by locality","b":true},{"text":" (everything from the header in one call, line items in another) rather than one call for forty fields. Long field lists degrade towards the end, and a call that produces eight fields well beats one that produces forty with the last ten drifting."}]},{"t":"h2","id":"the-line-items-problem","runs":[{"text":"The line-items problem"}]},{"t":"p","runs":[{"text":"Repeating rows are the hardest part of document extraction and the least discussed."}]},{"t":"p","runs":[{"text":"Models drop rows, merge rows that wrapped across a page break, and invent a row when a table has a subtotal line. The defences are arithmetic rather than linguistic:"}]},{"t":"code","lang":"python","label":"Python","text":"assert len(items) == stated_row_count          # ask for the count separately\nassert sum(i.amount_paise for i in items) == total_paise\nassert all(i.qty * i.unit_paise == i.amount_paise for i in items)"},{"t":"p","runs":[{"text":"Ask for the row count as its own field before the rows themselves, so you have two independent statements to compare. Any mismatch routes to review. This catches more real errors than any prompt improvement, and it costs three lines."}]},{"t":"h2","id":"batching-concurrency-and-cost","runs":[{"text":"Batching, concurrency and cost"}]},{"t":"p","runs":[{"text":"Ten thousand documents at one call each is a throughput problem with three levers."}]},{"t":"p","runs":[{"text":"Batch APIs.","b":true},{"text":" Most providers offer an asynchronous batch endpoint at roughly half price, with results in up to 24 hours. If the work is not interactive, and extraction rarely is, this halves the bill for no engineering effort beyond writing to a file and polling."}]},{"t":"p","runs":[{"text":"Concurrency with a cap.","b":true},{"text":" For interactive or same-day work, a semaphore of 5 to 20 in-flight requests, sized to your rate limit rather than your ambition. Above the limit you generate 429s, retry them, and go slower than if you had been patient."}]},{"t":"p","runs":[{"text":"Cheap model first.","b":true},{"text":" Extraction from clean text is exactly the task where a small model performs close to a large one. Route the documents that fail validation to the expensive model. On a typical corpus, 85 per cent never need it."}]},{"t":"p","runs":[{"text":"Always run a "},{"text":"pilot of a hundred documents","b":true},{"text":" before the full run. Measure accuracy, cost per document and the review rate, then multiply. A team that discovers at document 9,000 that the cost is four times the estimate has burned money it did not have to."}]},{"t":"h2","id":"the-review-queue-is-the-product","runs":[{"text":"The review queue is the product"}]},{"t":"p","runs":[{"text":"No extraction pipeline is right every time, so the real design question is not accuracy: it is what happens to the ones that are wrong."}]},{"t":"p","runs":[{"text":"Sort every result into three lanes:"}]},{"t":"ol","start":1,"items":[[{"text":"Passed everything","b":true},{"text":": schema valid, arithmetic consistent, quotes verified. Auto-accept."}],[{"text":"Failed a hard check","b":true},{"text":": totals do not add up, date out of range, required field null. Human review."}],[{"text":"Passed but flagged","b":true},{"text":": model marked itself low confidence, or the document type was unusual. Sampled review."}]]},{"t":"p","runs":[{"text":"Then instrument the queue. Two numbers matter and they are different: "},{"text":"review rate","b":true},{"text":" (what fraction of documents a person touches) and "},{"text":"override rate","b":true},{"text":" (what fraction of reviewed documents the person changes). A review rate of 20 per cent with an override rate of 2 per cent means you are wasting reviewer time on documents that were fine: tighten the flagging. An override rate of 40 per cent means your auto-accept lane is probably also wrong and you should widen review, not narrow it."}]},{"t":"p","runs":[{"text":"Make the review screen show the source region beside the extracted value. A reviewer who has to open the original PDF and search for the number reviews at a quarter of the speed, and the throughput of the whole pipeline is set by the human stage, not the model stage."}]},{"t":"h2","id":"the-corrections-are-the-asset","runs":[{"text":"The corrections are the asset"}]},{"t":"p","runs":[{"text":"Every override is a labelled example: the input, the wrong answer, the right answer. Store all three. After a few months you have a few thousand verified pairs, which is:"}]},{"t":"ul","items":[[{"text":"A serious evaluation set, drawn from exactly the distribution you serve"}],[{"text":"The training data for a distilled small model, if the volume justifies it"}],[{"text":"Direct evidence of which failure modes are common, which tells you what to fix next"}]]},{"t":"p","runs":[{"text":"Most teams throw this away because the reviewer’s correction is written straight into the database with no record of what it replaced. Capturing it costs one extra column and is the most valuable by-product the pipeline produces."}]}],"check":{"question":[{"text":"An invoice pipeline reports 97% field accuracy in testing but finance finds recurring errors in line items. Which change addresses the mechanism rather than the symptom?"}],"options":[{"text":[{"text":"Raise the model tier so line items are read more carefully."}],"correct":false,"tempting":[{"text":"Improves average reading without adding any way to detect a dropped or merged row, which is the specific failure with repeating structures."}]},{"text":[{"text":"Send each page separately so tables are never split across calls."}],"correct":false,"tempting":[{"text":"Helps with page-break merges but creates a new problem, since a table split across pages now has no call that sees the whole of it."}]},{"text":[{"text":"Ask for the row count as its own field and assert that the items sum to the stated total, routing any mismatch to review."}],"correct":true,"tempting":null},{"text":[{"text":"Increase the field-level accuracy target from 97% to 99% before shipping."}],"correct":false,"tempting":[{"text":"Sets a stricter goal without supplying a mechanism to reach it or to catch the residual errors that will still occur."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"schemas-models-can-follow","title":"Designing a schema a model can actually fill","minutes":9},"next":{"slug":"tool-use","title":"Tool use: the model never runs anything","minutes":8}}}