Structured output, and what a schema cannot promise

Stop parsing prose

The first version of every extraction feature asks for text and regexes it. It works on Monday. On Thursday the model writes “Sure, here’s the JSON:” before the JSON, and on Friday it wraps it in a code fence, and by the next week you have 200 lines of cleanup code that is a bad parser.

Ask for a schema instead. Every major provider supports either a JSON schema response format or tool-shaped output, where you declare a function signature and the model fills in the arguments. Both use constrained decoding: the model is prevented, token by token, from emitting anything the schema disallows.

ticket_schema = {
    "type": "object",
    "properties": {
        "category": {"type": "string",
                     "enum": ["refund", "schedule", "complaint", "other"]},
        "amount_minor_units": {"type": ["integer", "null"],
                               "description": "Amount in the smallest unit: paise, kobo, cents. Null if none stated."},
        "currency": {"type": ["string", "null"], "enum": ["NGN", "INR", "USD", "EUR", None]},
        "urgent": {"type": "boolean"},
    },
    "required": ["category", "urgent"],
    "additionalProperties": False,
}

Two choices in there are worth copying. Money is an integer in minor units, never a float: 4500 kobo, not 45.0. And every enum has an escape value, because a required enum with no way out forces the model to pick something.

Validate anyway

Constrained decoding is good, not perfect, and your schema will drift from your code. Parse into a typed object and let the failure be loud.

import json
from pydantic import BaseModel, ValidationError

class Ticket(BaseModel):
    category: str
    amount_minor_units: int | None = None
    currency: str | None = None
    urgent: bool

def extract(email: str, tries: int = 2) -> Ticket:
    messages = [{"role": "user", "content": email}]
    for _ in range(tries):
        raw = call_model_with_schema(messages, ticket_schema)
        try:
            return Ticket.model_validate(raw)
        except ValidationError as e:
            messages += [
                {"role": "assistant", "content": json.dumps(raw)},
                {"role": "user", "content": f"That failed validation:\n{e}\nReturn corrected JSON only."},
            ]
    raise RuntimeError("no valid extraction after retries")

Handing the validation error back is the cheapest repair loop there is, and it usually works on the first retry. Cap the retries. An unbounded repair loop is a bill.

The honest part: shape is not truth

Here is the thing teams miss, often for months. A schema guarantees that a field exists and has the right type. It says nothing about whether the value is correct.

{"amount_minor_units": 4500} is a perfectly valid object when the invoice said 450. Parse errors go to zero, so the feature looks fixed, and the wrong values now sail straight into your database with no exception to catch them.

Worse, required actively encourages this. A required field must be emitted. If the document does not contain an invoice number, the model has to write something, and something is what you get. This is one of the most reliable ways to manufacture hallucinations by accident.

So:

• Mark fields optional and nullable unless they truly must be present.
• Give every enum an other or unknown member.
• Add a source_quote field holding the exact text the value came from. It costs a few tokens and turns a silent wrong number into something you can check, by eye or in code.

"source_quote": {"type": ["string", "null"],
                 "description": "Exact substring of the input supporting the amount. Null if not stated."}

Then assert in code that source_quote is a substring of the input. That single check catches a surprising share of invented values, and it is deterministic: no second model call required.

Do not ask for confidence

A tempting field is "confidence": {"type": "number"}. Resist it. Self-reported confidence from a language model is poorly calibrated: it clusters around 0.9, and it is high on exactly the fluent, plausible errors you most want to catch. If you need a confidence signal, derive it from something real: did the quote match, do two runs agree, did retrieval find a supporting document.