Getting structure without training for it

Make invalid output impossible

Fine-tuning for format makes bad output rarer. Constrained decoding makes it impossible, and it needs no training at all.

The mechanism is simple and worth understanding rather than treating as magic. At every generation step the model produces a logit for each of 128,000 tokens. A constrained decoder maintains a state machine built from your schema, works out which tokens could legally come next given what has been generated, and sets every other logit to negative infinity before sampling.

If the schema says the next character must be " or }, then every token that does not begin with one of those has zero probability. The model cannot produce invalid JSON, not because it has learned not to, but because the invalid continuations were removed from the choice.

The tools, all free

from outlines import models, generate
from pydantic import BaseModel

class Triage(BaseModel):
    queue: Literal["billing","shipping","refunds","technical","other"]
    urgency: int
    summary: str

result = generate.json(models.transformers("Qwen/Qwen2.5-7B-Instruct"), Triage)(ticket)

That returns a validated object, always. Alternatives: llama.cpp supports GBNF grammars natively; vLLM has guided decoding built in; XGrammar and lm-format-enforcer provide the same capability with different performance characteristics. Nothing here costs money.

Grammars are more general than JSON schemas. Anything you can express as a context-free grammar (a SQL subset, a domain-specific language, a fixed report template, a limited set of tool calls) can be enforced the same way.

What it guarantees, and what it does not

It guarantees syntax. The output parses, the enum value is one of your five, the integer is an integer, the required fields are present.

It guarantees nothing about truth. A constrained decoder will happily emit {"queue": "billing", "urgency": 3} for a shipping complaint. Validity and correctness are different properties, and confusing them is the commonest mistake here.

There is also a real debate about whether heavy constraints reduce answer quality. The argument for: forcing a model into a rigid shape from the first token prevents it producing the intermediate reasoning it would otherwise use. The argument against: careful comparisons find little or no degradation when the schema is reasonable and the model is competent. Both positions have supporting experiments, the effect appears to depend heavily on the task and the schema, and it is not settled.

The pragmatic response is a design that sidesteps it: let the model reason freely in a reasoning string field that comes first in the schema, then constrain the structured fields that follow. You get the thinking and the guarantee.

The poor version, for hosted APIs

If you are calling an API that offers no grammar support, validate and retry:

for attempt in range(3):
    out = call_model(prompt)
    try:
        return Triage.model_validate_json(out)
    except ValidationError as e:
        prompt += f"\nThat was invalid: {e}. Return only valid JSON."
raise

This works, and it costs a full extra generation every time it fires. At a 4% invalid rate on a million requests that is 40,000 extra calls. Constrained decoding costs nothing per request; retries cost a generation per failure. That arithmetic is how you decide.

Many hosted providers now offer a structured-output mode that does the constraining server-side. Use it where it exists.

When fine-tuning for format is still right

Three cases survive:

1. The schema is enormous. A grammar with hundreds of fields makes the state machine large and the per-token masking expensive. A model that has learned the shape needs less help from the constraint.
2. The model must choose the schema. Constraining requires knowing which grammar to apply. If deciding which tool to call, or which of twelve document types this is, is itself the task, that decision cannot be constrained, though it can often be made by a cheap classifier, as module one argued.
3. The content inside the structure is wrong. If the JSON is valid and the values are poor, no constraint helps. That is a knowledge or capability problem, diagnosed as such.

The order to try things

1. Constrained decoding. Free, immediate, guaranteed.
2. Constrained decoding plus a few examples in the prompt, which improves the content inside the valid structure.
3. Fine-tuning, if the content is still wrong after both, and then keep the constraint as well, because a fine-tuned model still produces invalid output occasionally and the constraint costs nothing to keep.

That last point is the one to remember. These are not alternatives. Constrained decoding is a floor you should never remove, and fine-tuning raises what happens above it.