Structured extraction, where normalisation is most of the work

Three questions, three numbers

Pulling structured data out of documents (invoices, CVs, lab reports, product specs) fails at three levels, and merging them hides which one you have.

1. Did it produce valid output at all? Parses as JSON, matches the schema, required fields present, types correct. Measure this alone; it is usually fixable with constrained decoding or a retry, and it has nothing to do with reading the document.
2. Is each field right? Per-field precision and recall, reported per field, never averaged into one number. Invoice totals and supplier names have different difficulties and different consequences.
3. Is the whole record right? The number that matters if a human is not reviewing, and it is much lower than the field accuracy suggests: twelve fields at 97% each gives about 69% of records entirely correct.

Normalisation is where the honesty lives

Most of a first extraction evaluation measures the comparison code rather than the model. The gold file says 2026-03-31; the model says 31/03/2026. The gold says 120000; the model says 1,20,000. Both are right, and a naive string comparison scores both wrong.

Normalise both sides before comparing, and write the normaliser once:

def norm_amount(s):
    return round(float(re.sub(r"[^\d.]", "", s)), 2)

def norm_date(s):
    return dateparser.parse(s, settings={"DATE_ORDER": "DMY"}).date().isoformat()

Then report two field-accuracy numbers: exact string match and normalised match. The gap between them is a measure of your own formatting debt, and it is usually the cheapest thing on the list to fix.

Beware the reverse error: over-eager normalisation that lowercases and strips punctuation will happily score M/s Sharma & Co. equal to ms sharma co, which may be fine for a name and disastrous for a part number. Normalise per field type, not globally.

Missing and invented are different failures

Split the errors:

• Missing: gold has a value, the model returned null. Costs a human a lookup.
• Wrong: both non-null, values differ. Costs a wrong payment.
• Invented: model returned a value where the document has none. The most dangerous, because it looks like data.

Invention has a free detector. Any literal value the model extracted should appear in the source text after normalisation:

grounded = norm(value) in norm(source_text)

This catches the invoice number that was never printed, without a model call and without a gold label, which means you can run it on production traffic, not just on your labelled set. Fields that are computed rather than copied (a total the model summed) need an explicit exception, and that exception list is worth keeping short.

Write the field semantics down, or the labels will disagree

Extraction gold sets have a particular labelling problem: documents contain several plausible candidates. Three addresses appear on an invoice: registered, billing, delivery. Two dates: issue and due. Two names: the person and the company.

Annotators pick different ones, and the disagreement looks like model error later. The fix is in the schema, not the model: define each field by its role and its document cue, not by its type. “Supplier name: the entity issuing the invoice, usually beside the GSTIN in the header” is a definition two people can apply the same way.

Repeated fields need their own rule

Line items, previous employers, dependants: fields that appear many times per document. Comparing two lists needs a matching step before any scoring, and the choice of matching key changes the result. Match on a stable identifier where one exists, otherwise match greedily on the closest normalised description and report the matching rule alongside the score. Then compute precision and recall over line items, plus one count that users notice immediately: how often the number of rows was right.

Confidence, review rate, and the number the business wants

Extraction is where partial automation pays best, because a per-field confidence lets you auto-approve the easy documents and route the rest.

The metric that matters then is not accuracy at all. It is: at the field accuracy we require, what share of documents can go through untouched. A model at 94% accuracy with well-ordered confidence may auto-approve 70% of documents at 99.5% accuracy, which is a better product than a model at 96% accuracy with confidence scores that mean nothing. Measure the auto-approve rate at your required accuracy, and calibrate the confidence first, using the arithmetic from the calibration lesson.

Free tools cover all of this: pydantic or jsonschema for validity, dateparser and rapidfuzz for normalisation and near-matching, and a spreadsheet for the per-field breakdown if that is what your team will actually read.