Making the output valid by construction

Asking is not enough

Ask a 7B model to reply with JSON and it will, most of the time. The rest of the time it adds a sentence before the brace, wraps the object in a code fence, uses single quotes, or trails off. Retry loops and regular expressions to pull the object out are the usual response, and they are a bad answer to a problem with a good one.

Constrained decoding removes the failure entirely. At every generation step, a state machine tracks where you are in the grammar, works out which tokens could legally come next, and sets the logits of every other token to negative infinity before sampling. An invalid character is not unlikely; it is impossible.

Using it

llama.cpp, directly from a JSON Schema:

curl localhost:8080/v1/chat/completions -d '{
  "messages":[{"role":"user","content":"Extract from: Invoice 8841, 12 March, EUR 4,200, Meridian Ltd"}],
  "temperature":0,
  "response_format":{"type":"json_schema","json_schema":{"schema":{
    "type":"object",
    "properties":{
      "invoice_no":{"type":"string"},
      "date":{"type":"string","format":"date"},
      "currency":{"type":"string","enum":["EUR","USD","GBP","INR"]},
      "amount":{"type":"number"},
      "supplier":{"type":"string"}},
    "required":["invoice_no","currency","amount"],
    "additionalProperties":false}}}}'

vLLM takes guided_json, guided_choice and guided_regex. Outlines, xgrammar and lm-format-enforcer are the libraries underneath most of these, and can be used directly with transformers.

llama.cpp also accepts a raw grammar in its own GBNF notation, which is the escape hatch when a schema cannot express what you need:

root   ::= "SENTIMENT: " label "\nREASON: " [^\n]+ "\n"
label  ::= "positive" | "negative" | "neutral"

What it guarantees and what it does not

It guarantees syntax. The output will parse, the enum will be one of your values, the number will be a number.

It guarantees nothing about truth. A model forced to produce an amount will produce an amount, whether or not the document contains one. Constrained decoding converts a parse failure into a confidently wrong value, which is harder to notice. That trade is usually worth making, and you have to know you made it.

Three consequences follow.

Always give the model a way to say nothing. If a field may be absent, make it nullable or add an explicit "unknown" to the enum. A required field with no legal empty value is an instruction to fabricate.

Order the fields deliberately. Generation is left to right, so a field produced early conditions everything after it. Putting a short evidence or reasoning string before the answer field lets the model work before it commits; putting it after means the reasoning is a justification of a decision already made. This is a real and easily measured effect.

Keep schemas shallow. Deeply nested objects with many optional branches are harder for small models, and the grammar machinery gets slower to compile. Flat objects with clear names work far better at 7B than an elegant nested design.

Costs

Compiling a grammar takes time on first use, so cache it rather than rebuilding per request. Per-token overhead is small in modern implementations. The larger cost is subtler: a heavily constrained format can pull the model off the distribution it writes well in, so quality of the content can drop slightly even as validity goes to 100%. Measure both.

When not to constrain

Free-form prose, and any case where you would rather see the model fail visibly than produce a well-formed wrong answer. For a first exploration of a new task, run unconstrained and read what the model naturally produces; it tells you what it understood. Constrain afterwards, once you know the shape you want.

The practical upgrade path: prompt for JSON, discover the failure rate on 200 real inputs, then constrain and watch it go to zero. Skipping the measurement means you never find out whether the schema you wrote matches the task, because the errors that remain are now silent.