{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-extraction","title":"Structured extraction, where normalisation is most of the work","url":"https://addaly.com/learn/evaluating-ai/evaluating-extraction","position":33,"of":76,"minutes":9,"answer":"Score schema validity, field accuracy and whole-record correctness separately, normalise before comparing or you measure your own comparison code, and check every extracted value appears in the source to catch invention for free.","keyIdea":"Score schema validity, field accuracy and whole-record correctness separately, normalise before comparing or you measure your own comparison code, and check every extracted value appears in the source to catch invention for free."},"body":{"format":"markdown","text":"## Three questions, three numbers\n\nPulling structured data out of documents (invoices, CVs, lab reports, product specs) fails at three levels, and merging them hides which one you have.\n\n1. **Did it produce valid output at all?** Parses as JSON, matches the schema, required fields present, types correct. Measure this alone; it is usually fixable with constrained decoding or a retry, and it has nothing to do with reading the document.\n2. **Is each field right?** Per-field precision and recall, reported per field, never averaged into one number. Invoice totals and supplier names have different difficulties and different consequences.\n3. **Is the whole record right?** The number that matters if a human is not reviewing, and it is much lower than the field accuracy suggests: twelve fields at 97% each gives about 69% of records entirely correct.\n\n## Normalisation is where the honesty lives\n\nMost of a first extraction evaluation measures the comparison code rather than the model. The gold file says `2026-03-31`; the model says `31/03/2026`. The gold says `120000`; the model says `1,20,000`. Both are right, and a naive string comparison scores both wrong.\n\nNormalise both sides before comparing, and write the normaliser once:\n\n```python\ndef norm_amount(s):\n    return round(float(re.sub(r\"[^\\d.]\", \"\", s)), 2)\n\ndef norm_date(s):\n    return dateparser.parse(s, settings={\"DATE_ORDER\": \"DMY\"}).date().isoformat()\n```\n\nThen report two field-accuracy numbers: **exact** string match and **normalised** match. The gap between them is a measure of your own formatting debt, and it is usually the cheapest thing on the list to fix.\n\nBeware the reverse error: over-eager normalisation that lowercases and strips punctuation will happily score `M/s Sharma & Co.` equal to `ms sharma co`, which may be fine for a name and disastrous for a part number. Normalise per field type, not globally.\n\n## Missing and invented are different failures\n\nSplit the errors:\n\n- **Missing**: gold has a value, the model returned null. Costs a human a lookup.\n- **Wrong**: both non-null, values differ. Costs a wrong payment.\n- **Invented**: model returned a value where the document has none. The most dangerous, because it looks like data.\n\nInvention has a free detector. Any literal value the model extracted should appear in the source text after normalisation:\n\n```python\ngrounded = norm(value) in norm(source_text)\n```\n\nThis catches the invoice number that was never printed, without a model call and without a gold label, which means you can run it on production traffic, not just on your labelled set. Fields that are computed rather than copied (a total the model summed) need an explicit exception, and that exception list is worth keeping short.\n\n## Write the field semantics down, or the labels will disagree\n\nExtraction gold sets have a particular labelling problem: documents contain several plausible candidates. Three addresses appear on an invoice: registered, billing, delivery. Two dates: issue and due. Two names: the person and the company.\n\nAnnotators pick different ones, and the disagreement looks like model error later. The fix is in the schema, not the model: define each field by its role and its document cue, not by its type. \"Supplier name: the entity issuing the invoice, usually beside the GSTIN in the header\" is a definition two people can apply the same way.\n\n## Repeated fields need their own rule\n\nLine items, previous employers, dependants: fields that appear many times per document. Comparing two lists needs a matching step before any scoring, and the choice of matching key changes the result. Match on a stable identifier where one exists, otherwise match greedily on the closest normalised description and report the matching rule alongside the score. Then compute precision and recall over line items, plus one count that users notice immediately: how often the number of rows was right.\n\n## Confidence, review rate, and the number the business wants\n\nExtraction is where partial automation pays best, because a per-field confidence lets you auto-approve the easy documents and route the rest.\n\nThe metric that matters then is not accuracy at all. It is: **at the field accuracy we require, what share of documents can go through untouched.** A model at 94% accuracy with well-ordered confidence may auto-approve 70% of documents at 99.5% accuracy, which is a better product than a model at 96% accuracy with confidence scores that mean nothing. Measure the auto-approve rate at your required accuracy, and calibrate the confidence first, using the arithmetic from the calibration lesson.\n\nFree tools cover all of this: `pydantic` or `jsonschema` for validity, `dateparser` and `rapidfuzz` for normalisation and near-matching, and a spreadsheet for the per-field breakdown if that is what your team will actually read."},"check":{"question":"An extraction system scores 97% per-field accuracy across twelve fields, and the team promises fully automated processing. What should they check first?","options":[{"text":"Whether the fields are correlated, since correlated errors would make the record-level rate worse than independence implies.","correct":false,"tempting":"Correlation usually makes the record rate better than the independent estimate, since errors cluster in the same bad documents rather than spreading across records."},{"text":"The record-level rate, since twelve independent fields at 97% leaves only about 69% of documents entirely correct.","correct":true},{"text":"Whether the 97% was computed with exact or normalised matching, since the two can differ by several points.","correct":false,"tempting":"A real and useful check, but it moves the field number modestly; the automation claim founders on the multiplication across fields."},{"text":"The schema validity rate, since invalid JSON would not have been included in the field accuracy at all.","correct":false,"tempting":"Worth measuring separately, though invalid outputs are normally counted as failures rather than silently excluded, and it does not explain the automation gap."}]},"prev":{"slug":"evaluating-code-generation","title":"Code: run it, in a box, against tests it has not seen","url":"https://addaly.com/learn/evaluating-ai/evaluating-code-generation"},"next":{"slug":"evaluating-conversation","title":"Multi-turn: score the session, not the turn","url":"https://addaly.com/learn/evaluating-ai/evaluating-conversation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Structured extraction, where normalisation is most of the work', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-extraction (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Structured extraction, where normalisation is most of the work","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"Score schema validity, field accuracy and whole-record correctness separately, normalise before comparing or you measure your own comparison code, and check every extracted value appears in the source to catch invention for free.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-questions-three-numbers","runs":[{"text":"Three questions, three numbers"}]},{"t":"p","runs":[{"text":"Pulling structured data out of documents (invoices, CVs, lab reports, product specs) fails at three levels, and merging them hides which one you have."}]},{"t":"ol","start":1,"items":[[{"text":"Did it produce valid output at all?","b":true},{"text":" Parses as JSON, matches the schema, required fields present, types correct. Measure this alone; it is usually fixable with constrained decoding or a retry, and it has nothing to do with reading the document."}],[{"text":"Is each field right?","b":true},{"text":" Per-field precision and recall, reported per field, never averaged into one number. Invoice totals and supplier names have different difficulties and different consequences."}],[{"text":"Is the whole record right?","b":true},{"text":" The number that matters if a human is not reviewing, and it is much lower than the field accuracy suggests: twelve fields at 97% each gives about 69% of records entirely correct."}]]},{"t":"h2","id":"normalisation-is-where-the-honesty-lives","runs":[{"text":"Normalisation is where the honesty lives"}]},{"t":"p","runs":[{"text":"Most of a first extraction evaluation measures the comparison code rather than the model. The gold file says "},{"text":"2026-03-31","code":true},{"text":"; the model says "},{"text":"31/03/2026","code":true},{"text":". The gold says "},{"text":"120000","code":true},{"text":"; the model says "},{"text":"1,20,000","code":true},{"text":". Both are right, and a naive string comparison scores both wrong."}]},{"t":"p","runs":[{"text":"Normalise both sides before comparing, and write the normaliser once:"}]},{"t":"code","lang":"python","label":"Python","text":"def norm_amount(s):\n    return round(float(re.sub(r\"[^\\d.]\", \"\", s)), 2)\n\ndef norm_date(s):\n    return dateparser.parse(s, settings={\"DATE_ORDER\": \"DMY\"}).date().isoformat()"},{"t":"p","runs":[{"text":"Then report two field-accuracy numbers: "},{"text":"exact","b":true},{"text":" string match and "},{"text":"normalised","b":true},{"text":" match. The gap between them is a measure of your own formatting debt, and it is usually the cheapest thing on the list to fix."}]},{"t":"p","runs":[{"text":"Beware the reverse error: over-eager normalisation that lowercases and strips punctuation will happily score "},{"text":"M/s Sharma & Co.","code":true},{"text":" equal to "},{"text":"ms sharma co","code":true},{"text":", which may be fine for a name and disastrous for a part number. Normalise per field type, not globally."}]},{"t":"h2","id":"missing-and-invented-are-different-failures","runs":[{"text":"Missing and invented are different failures"}]},{"t":"p","runs":[{"text":"Split the errors:"}]},{"t":"ul","items":[[{"text":"Missing","b":true},{"text":": gold has a value, the model returned null. Costs a human a lookup."}],[{"text":"Wrong","b":true},{"text":": both non-null, values differ. Costs a wrong payment."}],[{"text":"Invented","b":true},{"text":": model returned a value where the document has none. The most dangerous, because it looks like data."}]]},{"t":"p","runs":[{"text":"Invention has a free detector. Any literal value the model extracted should appear in the source text after normalisation:"}]},{"t":"code","lang":"python","label":"Python","text":"grounded = norm(value) in norm(source_text)"},{"t":"p","runs":[{"text":"This catches the invoice number that was never printed, without a model call and without a gold label, which means you can run it on production traffic, not just on your labelled set. Fields that are computed rather than copied (a total the model summed) need an explicit exception, and that exception list is worth keeping short."}]},{"t":"h2","id":"write-the-field-semantics-down-or-the-labels-will-disagree","runs":[{"text":"Write the field semantics down, or the labels will disagree"}]},{"t":"p","runs":[{"text":"Extraction gold sets have a particular labelling problem: documents contain several plausible candidates. Three addresses appear on an invoice: registered, billing, delivery. Two dates: issue and due. Two names: the person and the company."}]},{"t":"p","runs":[{"text":"Annotators pick different ones, and the disagreement looks like model error later. The fix is in the schema, not the model: define each field by its role and its document cue, not by its type. “Supplier name: the entity issuing the invoice, usually beside the GSTIN in the header” is a definition two people can apply the same way."}]},{"t":"h2","id":"repeated-fields-need-their-own-rule","runs":[{"text":"Repeated fields need their own rule"}]},{"t":"p","runs":[{"text":"Line items, previous employers, dependants: fields that appear many times per document. Comparing two lists needs a matching step before any scoring, and the choice of matching key changes the result. Match on a stable identifier where one exists, otherwise match greedily on the closest normalised description and report the matching rule alongside the score. Then compute precision and recall over line items, plus one count that users notice immediately: how often the number of rows was right."}]},{"t":"h2","id":"confidence-review-rate-and-the-number-the-business-wants","runs":[{"text":"Confidence, review rate, and the number the business wants"}]},{"t":"p","runs":[{"text":"Extraction is where partial automation pays best, because a per-field confidence lets you auto-approve the easy documents and route the rest."}]},{"t":"p","runs":[{"text":"The metric that matters then is not accuracy at all. It is: "},{"text":"at the field accuracy we require, what share of documents can go through untouched.","b":true},{"text":" A model at 94% accuracy with well-ordered confidence may auto-approve 70% of documents at 99.5% accuracy, which is a better product than a model at 96% accuracy with confidence scores that mean nothing. Measure the auto-approve rate at your required accuracy, and calibrate the confidence first, using the arithmetic from the calibration lesson."}]},{"t":"p","runs":[{"text":"Free tools cover all of this: "},{"text":"pydantic","code":true},{"text":" or "},{"text":"jsonschema","code":true},{"text":" for validity, "},{"text":"dateparser","code":true},{"text":" and "},{"text":"rapidfuzz","code":true},{"text":" for normalisation and near-matching, and a spreadsheet for the per-field breakdown if that is what your team will actually read."}]}],"check":{"question":[{"text":"An extraction system scores 97% per-field accuracy across twelve fields, and the team promises fully automated processing. What should they check first?"}],"options":[{"text":[{"text":"Whether the fields are correlated, since correlated errors would make the record-level rate worse than independence implies."}],"correct":false,"tempting":[{"text":"Correlation usually makes the record rate better than the independent estimate, since errors cluster in the same bad documents rather than spreading across records."}]},{"text":[{"text":"The record-level rate, since twelve independent fields at 97% leaves only about 69% of documents entirely correct."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the 97% was computed with exact or normalised matching, since the two can differ by several points."}],"correct":false,"tempting":[{"text":"A real and useful check, but it moves the field number modestly; the automation claim founders on the multiplication across fields."}]},{"text":[{"text":"The schema validity rate, since invalid JSON would not have been included in the field accuracy at all."}],"correct":false,"tempting":[{"text":"Worth measuring separately, though invalid outputs are normally counted as failures rather than silently excluded, and it does not explain the automation gap."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"evaluating-code-generation","title":"Code: run it, in a box, against tests it has not seen","minutes":9},"next":{"slug":"evaluating-conversation","title":"Multi-turn: score the session, not the turn","minutes":9}}}