Designing a schema a model can actually fill

Two schemas, same information

JSON Schema lets you express things a model will not reliably produce. The schema below is valid and will disappoint you:

{"type":"object","properties":{
  "invoice":{"type":"object","properties":{
    "parties":{"type":"object","properties":{
      "vendor":{"type":"object","properties":{
        "registration":{"type":"object","properties":{
          "gstin":{"type":"string","pattern":"^[0-9]{2}[A-Z]{5}[0-9]{4}[A-Z]{1}[1-9A-Z]{1}Z[0-9A-Z]{1}$"}
}}}}}}}}}}

This one carries the same information and works:

{"type":"object","properties":{
  "vendor_name":   {"type":["string","null"]},
  "vendor_gstin":  {"type":["string","null"]},
  "invoice_date":  {"type":["string","null"], "description":"YYYY-MM-DD, or null if not legible"},
  "total_paise":   {"type":["integer","null"]},
  "confidence":    {"enum":["high","low"]}
},"required":["vendor_name","vendor_gstin","invoice_date","total_paise","confidence"]}

Five rules produced that difference, and they hold across providers.

The five rules

Flat beats nested. Every level of nesting is another chance to close a brace early or forget a key. Three levels is a reasonable ceiling. Deep hierarchies are for storage; flatten for the model and reassemble in your code, where nesting is free.

Enums beat free text. "status": "high" from a three-value enum is a decision. "status": "fairly confident" is prose you will end up parsing with a regular expression. Any field with a known set of values should be an enum, and the set should be short: beyond about a dozen options, accuracy falls and you are better off splitting the decision.

Explicit null beats omission. If a field can be absent, make it nullable and required rather than optional. A required nullable field forces the model to make a decision it can be graded on; an optional field lets it skip the hard one, and your downstream code cannot tell “not present” from “not found”.

Name the unit in the field name. total_paise, duration_seconds, weight_grams. Models produce "total": 1499.00 and you will never know whether that was rupees or paise until an accountant tells you. Integers in the smallest unit also avoid float rounding.

Descriptions are prompt text. The description of a field is read by the model and is often the most efficient place to put a rule, because it sits next to the thing it governs. "description": "YYYY-MM-DD, or null if not legible" does more work than the same sentence buried in a 900-token system prompt.

Constrained decoding, and what it does not guarantee

Several providers offer strict structured output, and the mechanism is worth knowing. At each generation step the schema is compiled into a state machine over the token vocabulary, and tokens that cannot continue a valid document have their logits set to negative infinity before sampling. The model is not persuaded to produce valid JSON; it is made incapable of producing anything else.

When it is available, use it: it eliminates an entire category of parse failure. Three caveats:

• Support is partial. Complex regular expressions, oneOf with overlapping branches, recursive definitions and some numeric constraints are often unsupported, and the API tells you at request time, not before.
• The first request compiles the grammar, which adds latency; later requests with the same schema reuse it.
• It constrains shape, never content. Constrained decoding will happily emit "invoice_date": "2024-13-45" if the schema says string, or a perfectly formatted GSTIN that does not belong to the vendor. This is the same honest limit from the structured-output lesson, and constrained decoding does not soften it at all: it makes the wrong answer syntactically flawless.

Where it is unavailable: prefill the assistant turn with {, supply one example object, set temperature to 0, and validate with a retry that includes the validator’s error message. That combination gets you into the high nineties without any special support.

Validate with a real validator

from pydantic import BaseModel, Field, ValidationError
from typing import Literal, Optional

class Invoice(BaseModel):
    vendor_name: Optional[str]
    vendor_gstin: Optional[str] = Field(pattern=GSTIN_RE)
    invoice_date: Optional[str]
    total_paise: Optional[int] = Field(ge=0)
    confidence: Literal["high", "low"]

try:
    inv = Invoice.model_validate_json(raw)
except ValidationError as e:
    raw = retry(prompt, hint=str(e))    # give the model the error

Feeding the validation error back is the single highest-yield retry in this whole area. total_paise: Input should be a valid integer is far more actionable to a model than “try again”, and one retry with the error attached typically fixes the large majority of remaining failures.

Keep the retry count at one or two. A model that has failed the schema twice with the error in front of it is not going to succeed on the fifth attempt, and you are now paying five times for one record.

Add an abstention field

The most useful field in most extraction schemas is not data. It is confidence, restricted to two or three values, with the criterion written in the description: low if the field was inferred rather than read, or if the scan was unclear.

This is not the model’s probability of being right: nothing in the response gives you that. It is a self-report, and it is weakly correlated with correctness. But a binary self-report with a clear criterion is good enough to route: everything marked low goes to a human queue. You have converted an unknown error rate into a known review rate, which is a thing you can staff and measure.