{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"extraction-that-holds-up","title":"Turning a document into a row you can defend","url":"https://addaly.com/learn/agents-and-automation/extraction-that-holds-up","position":23,"of":72,"minutes":10,"answer":"Make the model quote its evidence and check the quote appears verbatim in the source, which converts the dangerous failure (a confident wrong value) into the safe one, a blank field.","keyIdea":"Make the model quote its evidence and check the quote appears verbatim in the source, which converts the dangerous failure (a confident wrong value) into the safe one, a blank field."},"body":{"format":"markdown","text":"## Two kinds of error, and only one of them hurts\n\nAn extractor makes two mistakes. It misses a field that was there, or it fills a field with something that was not. The first is annoying and visible: a blank appears, somebody looks at the document. The second is a wrong number entering a ledger with no signal attached, and it can survive for a year.\n\nThe whole design goal is therefore to **convert wrong answers into missing ones**. Everything below serves that.\n\n## The quote check\n\nRequire the model to return, for every field, a verbatim snippet of the source text containing the value:\n\n```json\n{\n  \"invoice_total\": \"48,200.00\",\n  \"invoice_total_quote\": \"Total payable  48,200.00\",\n  \"invoice_total_page\": 2\n}\n```\n\nThen, in code, search the extracted text of page 2 for that quote. Not with a model: with a string search, after collapsing whitespace. If the quote is not there, the field is rejected and set to null with a reason.\n\nThis is the strongest cheap check in document extraction, and its power comes from where it sits. Fluency cannot get past it. A model that invents a total must also invent a line of source text, and the invented line will not be present in the document. In practice a large share of hallucinated fields fail this test immediately, and the ones that pass are usually genuine values that were misread rather than imagined: a different and more tractable problem.\n\nAllow for OCR noise by comparing loosely: strip spaces and punctuation, and accept a near match above a high similarity threshold. Do not loosen it further than that, or you have reinvented the problem.\n\n## Make absence expressible\n\nAs covered in the schema lesson: if a field is required and the value is not on the page, the model will produce something. So every field gets a status (`printed`, `inferred`, or `absent`) and the pipeline treats them differently. Inferred values are usable for triage and never for payment.\n\n## Extract as printed, normalise afterwards\n\nPull `\"1,20,500.00\"` and `\"31 Mar 26\"` exactly as they appear, then convert in your own code, in a separate step, with the raw string retained beside the parsed value.\n\nDoing both at once looks tidier and destroys your ability to debug. When a total comes out a hundred times too large, you need to know whether the model misread the page or your parser mishandled a separator, and if the printed string was never stored you cannot tell.\n\n## Measure per field, never in aggregate\n\nLabel fifty to a hundred real documents by hand, an afternoon, and the highest-value afternoon in the project, and score each field separately.\n\nYou will find something like: supplier name 99 per cent, invoice number 97, total 96, due date 71, line items 62. A single \"94 per cent accurate\" headline hides that, and the due date is the field that decides when money moves. Per-field numbers also tell you where to spend: the fix for line items is rarely the fix for dates.\n\n> Figure: One hundred hand-checked invoices, scored field by field. Bars (% of documents where the field was right): Supplier name 99; Invoice number 97; Total 96; Due date 71; Line items 62. The single headline for this extractor is eighty-five per cent, and it is useless. The due date decides when money moves and is wrong nearly a third of the time, and the fix for line items is not the fix for dates.\n\nKeep those documents as a fixed set and re-run it whenever you change the prompt, the model or the parser. This is the regression test for the part of the system that has no unit tests.\n\n## Design the human step, not just the model step\n\nNobody automates document extraction to 100 per cent. The realistic target is to route the uncertain fraction to a person efficiently.\n\nShow them the page image with the quote highlighted and the extracted value in an editable box beside it. That is a five to twenty second decision. Sending them a PDF and a spreadsheet and asking them to check is a four-minute decision, and the difference between those two numbers is the difference between a project that saves time and one that moves it.\n\nSend to review: anything with a failed quote check, anything marked inferred, anything where the arithmetic does not reconcile, and a random one in twenty of everything else. The random sample is not optional: it is the only way to learn the error rate among cases the system was confident about.\n\n## Multi-page and amended values\n\nA field can appear more than once with different values: a total on page one and a revised total on page four, or an amendment stapled to the front. Decide the rule explicitly (usually the latest occurrence wins, or the one on the page carrying the word \"revised\") record which occurrence was used, and flag the disagreement rather than silently choosing. A silent choice between two totals is exactly the kind of decision nobody knows was made until an auditor asks."},"check":{"question":"A team adds a rule requiring the model to return a verbatim quote for every extracted field, which they then search for in the source text. Why does this catch fabricated values that a second model call reviewing the output would not?","options":[{"text":"It runs faster, so the check can be applied to every field rather than a sample.","correct":false,"tempting":"Speed makes it affordable, but a cheap check that could still be fooled by fluent text would not add safety at any price."},{"text":"A second model call would be biased by seeing the first model’s answer, so removing that context restores its judgement.","correct":false,"tempting":"Anchoring is real, but a blind reviewer model still judges plausibility rather than presence, which is the property that matters here."},{"text":"Quoting forces the model to attend to the source text, which improves its accuracy at extraction time.","correct":false,"tempting":"Grounding may help a little, but the value is in the mechanical verification afterwards, not in the effect on generation."},{"text":"It moves the test outside the model: a string search either finds the line in the document or does not, and fluent invention cannot satisfy it.","correct":true}]},"prev":{"slug":"parsing-real-documents","title":"PDFs, scans and the layout problem","url":"https://addaly.com/learn/agents-and-automation/parsing-real-documents"},"next":{"slug":"dates-names-and-numbers","title":"The boring bugs that eat automations","url":"https://addaly.com/learn/agents-and-automation/dates-names-and-numbers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Turning a document into a row you can defend', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/extraction-that-holds-up (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Turning a document into a row you can defend","courseTitle":"Agents and Automation","module":{"index":3,"slug":"data-plumbing","title":"The data underneath: sources, documents, records"},"answer":null,"keyIdea":"Make the model quote its evidence and check the quote appears verbatim in the source, which converts the dangerous failure (a confident wrong value) into the safe one, a blank field.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of documents where the field was right","title":"One hundred hand-checked invoices, scored field by field","series":[{"label":"Supplier name","value":99},{"label":"Invoice number","value":97},{"label":"Total","value":96},{"label":"Due date","value":71,"highlight":true},{"label":"Line items","value":62}],"caption":"The single headline for this extractor is eighty-five per cent, and it is useless. The due date decides when money moves and is wrong nearly a third of the time, and the fix for line items is not the fix for dates."},"alt":"Figure: One hundred hand-checked invoices, scored field by field. Bars (% of documents where the field was right): Supplier name 99; Invoice number 97; Total 96; Due date 71; Line items 62. The single headline for this extractor is eighty-five per cent, and it is useless. The due date decides when money moves and is wrong nearly a third of the time, and the fix for line items is not the fix for dates."}},"blocks":[{"t":"h2","id":"two-kinds-of-error-and-only-one-of-them-hurts","runs":[{"text":"Two kinds of error, and only one of them hurts"}]},{"t":"p","runs":[{"text":"An extractor makes two mistakes. It misses a field that was there, or it fills a field with something that was not. The first is annoying and visible: a blank appears, somebody looks at the document. The second is a wrong number entering a ledger with no signal attached, and it can survive for a year."}]},{"t":"p","runs":[{"text":"The whole design goal is therefore to "},{"text":"convert wrong answers into missing ones","b":true},{"text":". Everything below serves that."}]},{"t":"h2","id":"the-quote-check","runs":[{"text":"The quote check"}]},{"t":"p","runs":[{"text":"Require the model to return, for every field, a verbatim snippet of the source text containing the value:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\n  \"invoice_total\": \"48,200.00\",\n  \"invoice_total_quote\": \"Total payable  48,200.00\",\n  \"invoice_total_page\": 2\n}"},{"t":"p","runs":[{"text":"Then, in code, search the extracted text of page 2 for that quote. Not with a model: with a string search, after collapsing whitespace. If the quote is not there, the field is rejected and set to null with a reason."}]},{"t":"p","runs":[{"text":"This is the strongest cheap check in document extraction, and its power comes from where it sits. Fluency cannot get past it. A model that invents a total must also invent a line of source text, and the invented line will not be present in the document. In practice a large share of hallucinated fields fail this test immediately, and the ones that pass are usually genuine values that were misread rather than imagined: a different and more tractable problem."}]},{"t":"p","runs":[{"text":"Allow for OCR noise by comparing loosely: strip spaces and punctuation, and accept a near match above a high similarity threshold. Do not loosen it further than that, or you have reinvented the problem."}]},{"t":"h2","id":"make-absence-expressible","runs":[{"text":"Make absence expressible"}]},{"t":"p","runs":[{"text":"As covered in the schema lesson: if a field is required and the value is not on the page, the model will produce something. So every field gets a status ("},{"text":"printed","code":true},{"text":", "},{"text":"inferred","code":true},{"text":", or "},{"text":"absent","code":true},{"text":") and the pipeline treats them differently. Inferred values are usable for triage and never for payment."}]},{"t":"h2","id":"extract-as-printed-normalise-afterwards","runs":[{"text":"Extract as printed, normalise afterwards"}]},{"t":"p","runs":[{"text":"Pull "},{"text":"\"1,20,500.00\"","code":true},{"text":" and "},{"text":"\"31 Mar 26\"","code":true},{"text":" exactly as they appear, then convert in your own code, in a separate step, with the raw string retained beside the parsed value."}]},{"t":"p","runs":[{"text":"Doing both at once looks tidier and destroys your ability to debug. When a total comes out a hundred times too large, you need to know whether the model misread the page or your parser mishandled a separator, and if the printed string was never stored you cannot tell."}]},{"t":"h2","id":"measure-per-field-never-in-aggregate","runs":[{"text":"Measure per field, never in aggregate"}]},{"t":"p","runs":[{"text":"Label fifty to a hundred real documents by hand, an afternoon, and the highest-value afternoon in the project, and score each field separately."}]},{"t":"p","runs":[{"text":"You will find something like: supplier name 99 per cent, invoice number 97, total 96, due date 71, line items 62. A single “94 per cent accurate” headline hides that, and the due date is the field that decides when money moves. Per-field numbers also tell you where to spend: the fix for line items is rarely the fix for dates."}]},{"t":"p","runs":[{"text":"Keep those documents as a fixed set and re-run it whenever you change the prompt, the model or the parser. This is the regression test for the part of the system that has no unit tests."}]},{"t":"h2","id":"design-the-human-step-not-just-the-model-step","runs":[{"text":"Design the human step, not just the model step"}]},{"t":"p","runs":[{"text":"Nobody automates document extraction to 100 per cent. The realistic target is to route the uncertain fraction to a person efficiently."}]},{"t":"p","runs":[{"text":"Show them the page image with the quote highlighted and the extracted value in an editable box beside it. That is a five to twenty second decision. Sending them a PDF and a spreadsheet and asking them to check is a four-minute decision, and the difference between those two numbers is the difference between a project that saves time and one that moves it."}]},{"t":"p","runs":[{"text":"Send to review: anything with a failed quote check, anything marked inferred, anything where the arithmetic does not reconcile, and a random one in twenty of everything else. The random sample is not optional: it is the only way to learn the error rate among cases the system was confident about."}]},{"t":"h2","id":"multi-page-and-amended-values","runs":[{"text":"Multi-page and amended values"}]},{"t":"p","runs":[{"text":"A field can appear more than once with different values: a total on page one and a revised total on page four, or an amendment stapled to the front. Decide the rule explicitly (usually the latest occurrence wins, or the one on the page carrying the word “revised”) record which occurrence was used, and flag the disagreement rather than silently choosing. A silent choice between two totals is exactly the kind of decision nobody knows was made until an auditor asks."}]}],"check":{"question":[{"text":"A team adds a rule requiring the model to return a verbatim quote for every extracted field, which they then search for in the source text. Why does this catch fabricated values that a second model call reviewing the output would not?"}],"options":[{"text":[{"text":"It runs faster, so the check can be applied to every field rather than a sample."}],"correct":false,"tempting":[{"text":"Speed makes it affordable, but a cheap check that could still be fooled by fluent text would not add safety at any price."}]},{"text":[{"text":"A second model call would be biased by seeing the first model’s answer, so removing that context restores its judgement."}],"correct":false,"tempting":[{"text":"Anchoring is real, but a blind reviewer model still judges plausibility rather than presence, which is the property that matters here."}]},{"text":[{"text":"Quoting forces the model to attend to the source text, which improves its accuracy at extraction time."}],"correct":false,"tempting":[{"text":"Grounding may help a little, but the value is in the mechanical verification afterwards, not in the effect on generation."}]},{"text":[{"text":"It moves the test outside the model: a string search either finds the line in the document or does not, and fluent invention cannot satisfy it."}],"correct":true,"tempting":null}],"order":[2,0,1,3],"why":null},"prev":{"slug":"parsing-real-documents","title":"PDFs, scans and the layout problem","minutes":10},"next":{"slug":"dates-names-and-numbers","title":"The boring bugs that eat automations","minutes":9}}}