Output that parses every time

Asking politely does not scale

“Respond only with JSON” works most of the time on a good model, and most of the time is the problem. A small model will occasionally wrap the JSON in prose, add a trailing comma, use single quotes, or begin with a markdown fence. At one failure in fifty, a pipeline processing ten thousand documents produces two hundred exceptions.

Local inference has a better answer than prompting, and it is one of the genuine advantages of running the model yourself: you control the sampler, so you can make invalid output impossible rather than unlikely.

Constrained decoding

At every step the model produces a distribution over the whole vocabulary. A grammar engine computes which tokens could legally come next given the output so far, sets every other token’s probability to zero, and samples from what remains.

If the schema says the next character must be " or }, no other token can be chosen. The JSON is valid by construction, not by good behaviour.

# llama.cpp with a JSON schema
curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role":"user","content":"Extract name and age."}],
  "response_format": {
    "type": "json_schema",
    "json_schema": {"schema": {
      "type": "object",
      "properties": {"name": {"type":"string"}, "age": {"type":"integer"}},
      "required": ["name", "age"]
    }}
  }
}'

llama.cpp also accepts GBNF grammars directly, which express things a JSON schema cannot:

root   ::= answer
answer ::= "yes" | "no" | "unclear"

That grammar makes any other output impossible. For classification into a fixed label set, this is the correct tool and it removes an entire category of parsing code.

vLLM and SGLang offer the same through their own backends, and SGLang’s implementation is notably fast because it precompiles the schema into a token mask.

The cost nobody mentions

Constrained decoding is not free, and the trade is worth understanding.

It can reduce answer quality. Forcing the model down a syntactic path prevents it from reasoning first. A model asked to output {"answer": immediately has to commit before thinking. The fix is structural: let it produce a short free-text reasoning field first in the schema, then the answer. Field order in your schema is generation order, and it matters.

Over-constraining hurts. A schema with fifteen required fields and strict enumerations pushes the model into a shape it may not have anything sensible to put in. It will fill the fields anyway, because it must. Valid output is not correct output, and a schema guarantees only the first.

There is a speed cost, small in good implementations and noticeable in poor ones, because the legal-token mask has to be computed at every step.

Tokenisation makes it fiddly. Grammars operate on characters and models emit tokens that may straddle boundaries, so implementations must handle partial matches. This is why grammar support was slow to arrive and why edge cases still exist with unusual vocabularies.

Practical design

1. Keep schemas small. Three to six fields. Run two passes rather than one enormous schema.
2. Put a reasoning field first if the task needs any judgement.
3. Use enumerations for anything categorical. They eliminate spelling variation and near-synonyms at a stroke.
4. Avoid deeply nested objects. Small models handle flat structures far more reliably.
5. Validate anyway. The grammar guarantees the shape; your code should still check that the age is plausible and the date exists.

When not to constrain

If the model needs to say “I cannot answer this from the given text”, a rigid schema removes its ability to. Include an explicit escape (a "status" field with an "insufficient_information" value) or you have built a machine that fabricates whenever the answer is absent, and it will.

That is the honest limitation of the whole technique. Constrained decoding guarantees the output parses. It says nothing whatever about whether the content is true, and by removing the model’s ability to decline, a careless schema makes fabrication more likely rather than less.