How a schema is actually enforced

Masking, not asking

When you set a response format and get back valid JSON every time, no part of that reliability came from the model deciding to cooperate. It came from arithmetic performed after the model spoke and before the token was chosen.

At every decoding step the model produces a logit for each token in its vocabulary: perhaps 130,000 numbers. Ordinarily the sampler picks from that distribution. Under constrained decoding, a separate component first computes which tokens could legally continue the output given the grammar and what has been emitted so far, and sets the logit of every other token to negative infinity. After the softmax, those tokens have probability zero. They cannot be sampled. Not unlikely: impossible.

If the schema says the next thing must be one of three enum strings and the model has already emitted "status": ", then the only permitted next tokens are those that begin one of those three strings. The model’s preference among the three still decides the outcome. Its preference for a fourth word it invented does not get a vote.

The machinery that computes the legal set is a state machine compiled from the schema: a finite automaton for regular structures, a pushdown automaton where nesting has to be tracked. You can read the implementations: outlines and xgrammar in Python, GBNF grammars in llama.cpp. All free, all local, all runnable on a laptop, and reading one for twenty minutes is worth more than any amount of speculation about what “JSON mode” does.

What this buys, precisely

It buys syntax. The output will parse. Required fields will be present. An enum field will hold one of the enum values. A field typed as integer will contain digits.

That is a genuine and large win. Before constrained decoding, the standard production pattern was: call, try to parse, fail on roughly one call in fifty, retry with the error appended, and carry the retry cost and latency forever. That failure mode is gone.

It is also the entire list. Consider what the guarantee does not touch:

• The integer can be wrong.
• The date can parse as a date and describe a day the event did not happen.
• The citations array can be present, correctly typed, and contain identifiers for passages that were never in the window.
• A field typed as string can contain an apology.

The schema is a constraint on the shape of the container. Nothing about the container constrains its contents.

The enum trap

This one is worth its own heading because it is the most common way a schema makes a system quietly worse.

Suppose you classify support messages into refund, replacement, escalate. The schema constrains the output to those three. Now a customer writes to ask what time you close.

The model cannot say “none of these”. The tokens for that answer have probability zero. It will emit one of the three, and it will emit it with no hedging, because the format gives it nowhere to hedge. Downstream, your code reads escalate and creates a ticket. The confidence you perceive in that output is an artefact of the mask.

The fix is in the schema, not the prompt:

"intent": {
  "type": "string",
  "enum": ["refund", "replacement", "escalate", "other", "unclear"]
}

other for messages that are about something else, unclear for messages too ambiguous to classify. Both routed to a human. Then measure how often each fires. If other never fires on real traffic, your enum is probably fine or your model is ignoring the escape hatch, and you can tell which by sampling. If it fires on eight percent, you have just discovered eight percent of your traffic that used to be silently misrouted.

Every closed set of options needs an explicit escape. This applies to boolean fields too: a required boolean forces a claim where the honest answer is that the document does not say.

The costs, which are small but real

Compilation. The automaton is built from the schema. For a simple flat object this is a millisecond. For deeply nested schemas with many optional branches and regular-expression constraints it can reach hundreds of milliseconds. It is cacheable per schema, and every serious implementation caches it; but a system that generates a fresh schema per request defeats that cache and pays the build every time.

Per-token overhead. Computing and applying the mask costs time at each step. Modern implementations do it concurrently with the forward pass and the overhead is usually under a few percent. It is not zero.

Quality, and this is the one to watch. Forcing structure from the first token removes the model’s room to work. If the schema’s first field is answer, the model must commit before it has reasoned. On tasks that need any intermediate steps, this measurably degrades accuracy.

The remedy is to give the reasoning a field of its own and put it first:

{
  "type": "object",
  "properties": {
    "evidence": {"type": "string", "description": "Quote the passage this rests on."},
    "reasoning": {"type": "string"},
    "answer": {"type": "string"}
  },
  "required": ["evidence", "reasoning", "answer"]
}

JSON object fields are generated in schema order. Put the thinking before the conclusion and the conclusion is conditioned on it. Put it after and it is a justification written for a decision already made, which is a different and much less useful thing, and reads identically.

Where the ground is not settled

Whether constrained decoding degrades reasoning even when the field order is sensible is still argued. Some studies report a real drop on reasoning benchmarks under strict schemas; others attribute most of the gap to prompt and ordering differences rather than to masking itself. The honest position is that the effect exists, its size depends on model and task, and it is small enough that you will not notice it without measuring.

So measure it. Run your golden set free-form with the schema described in prose, and again under enforcement, and compare accuracy on the field that matters. If enforcement costs you two points of accuracy and saves a retry path, that is a trade you can now actually make, instead of guessing.