{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"structured-output","title":"Making the output valid by construction","url":"https://addaly.com/learn/open-models/structured-output","position":42,"of":71,"minutes":9,"answer":"Grammar-constrained decoding masks illegal tokens at each step so invalid syntax becomes impossible, which converts parse failures into silently wrong values unless the schema gives the model an explicit way to express absence.","keyIdea":"Grammar-constrained decoding masks illegal tokens at each step so invalid syntax becomes impossible, which converts parse failures into silently wrong values unless the schema gives the model an explicit way to express absence."},"body":{"format":"markdown","text":"## Asking is not enough\n\nAsk a 7B model to reply with JSON and it will, most of the time. The rest of the time it adds a sentence before the brace, wraps the object in a code fence, uses single quotes, or trails off. Retry loops and regular expressions to pull the object out are the usual response, and they are a bad answer to a problem with a good one.\n\nConstrained decoding removes the failure entirely. At every generation step, a state machine tracks where you are in the grammar, works out which tokens could legally come next, and sets the logits of every other token to negative infinity before sampling. An invalid character is not unlikely; it is impossible.\n\n## Using it\n\n**llama.cpp**, directly from a JSON Schema:\n\n```bash\ncurl localhost:8080/v1/chat/completions -d '{\n  \"messages\":[{\"role\":\"user\",\"content\":\"Extract from: Invoice 8841, 12 March, EUR 4,200, Meridian Ltd\"}],\n  \"temperature\":0,\n  \"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"schema\":{\n    \"type\":\"object\",\n    \"properties\":{\n      \"invoice_no\":{\"type\":\"string\"},\n      \"date\":{\"type\":\"string\",\"format\":\"date\"},\n      \"currency\":{\"type\":\"string\",\"enum\":[\"EUR\",\"USD\",\"GBP\",\"INR\"]},\n      \"amount\":{\"type\":\"number\"},\n      \"supplier\":{\"type\":\"string\"}},\n    \"required\":[\"invoice_no\",\"currency\",\"amount\"],\n    \"additionalProperties\":false}}}}'\n```\n\n**vLLM** takes `guided_json`, `guided_choice` and `guided_regex`. **Outlines**, **xgrammar** and **lm-format-enforcer** are the libraries underneath most of these, and can be used directly with `transformers`.\n\nllama.cpp also accepts a raw grammar in its own GBNF notation, which is the escape hatch when a schema cannot express what you need:\n\n```\nroot   ::= \"SENTIMENT: \" label \"\\nREASON: \" [^\\n]+ \"\\n\"\nlabel  ::= \"positive\" | \"negative\" | \"neutral\"\n```\n\n## What it guarantees and what it does not\n\nIt guarantees **syntax**. The output will parse, the enum will be one of your values, the number will be a number.\n\nIt guarantees nothing about **truth**. A model forced to produce an amount will produce an amount, whether or not the document contains one. Constrained decoding converts a parse failure into a confidently wrong value, which is harder to notice. That trade is usually worth making, and you have to know you made it.\n\n> Figure: What constrained decoding buys, and what it does not. A two by two grid, rows Prompted for JSON and Grammar-constrained, columns The output parses and The output is true: Prompted for JSON and The output parses: Most of the time; Prompted for JSON and The output is true: Unchanged; Grammar-constrained and The output parses: Always; Grammar-constrained and The output is true: Unchanged. Constraining converts a parse failure into a confidently wrong value, which is harder to notice. The trade is usually worth making and you have to know you made it: make the field nullable, or add an explicit unknown to the enum, so the model has somewhere to go.\n\nThree consequences follow.\n\n**Always give the model a way to say nothing.** If a field may be absent, make it nullable or add an explicit `\"unknown\"` to the enum. A required field with no legal empty value is an instruction to fabricate.\n\n**Order the fields deliberately.** Generation is left to right, so a field produced early conditions everything after it. Putting a short `evidence` or `reasoning` string before the answer field lets the model work before it commits; putting it after means the reasoning is a justification of a decision already made. This is a real and easily measured effect.\n\n**Keep schemas shallow.** Deeply nested objects with many optional branches are harder for small models, and the grammar machinery gets slower to compile. Flat objects with clear names work far better at 7B than an elegant nested design.\n\n## Costs\n\nCompiling a grammar takes time on first use, so cache it rather than rebuilding per request. Per-token overhead is small in modern implementations. The larger cost is subtler: a heavily constrained format can pull the model off the distribution it writes well in, so quality of the *content* can drop slightly even as validity goes to 100%. Measure both.\n\n## When not to constrain\n\nFree-form prose, and any case where you would rather see the model fail visibly than produce a well-formed wrong answer. For a first exploration of a new task, run unconstrained and read what the model naturally produces; it tells you what it understood. Constrain afterwards, once you know the shape you want.\n\n> The practical upgrade path: prompt for JSON, discover the failure rate on 200 real inputs, then constrain and watch it go to zero. Skipping the measurement means you never find out whether the schema you wrote matches the task, because the errors that remain are now silent."},"check":{"question":"After enabling JSON Schema constrained decoding, an extraction pipeline's parse errors drop to zero but downstream data quality gets worse. What is the most likely mechanism?","options":[{"text":"Constrained decoding lowers the model's effective temperature, reducing the diversity needed for accurate extraction.","correct":false,"tempting":"Masking removes illegal tokens rather than changing the temperature applied to the legal ones."},{"text":"Required fields with no null or unknown option force the model to emit a value even when the document contains none, so failures become plausible wrong answers instead of errors.","correct":true},{"text":"The grammar compilation adds latency, causing timeouts that truncate results.","correct":false,"tempting":"Compilation is a one-off cost and would produce missing responses, not lower-quality complete ones."},{"text":"JSON output uses more tokens than free text, so the model runs out of context.","correct":false,"tempting":"Structured output is usually shorter than prose, and truncation would show as parse failures, which have gone to zero."}]},"prev":{"slug":"sampling-parameters","title":"The sampler, and the defaults that ship broken","url":"https://addaly.com/learn/open-models/sampling-parameters"},"next":{"slug":"prompting-open-models","title":"What changes when nothing is hidden","url":"https://addaly.com/learn/open-models/prompting-open-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Making the output valid by construction', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/structured-output (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Making the output valid by construction","courseTitle":"The Open Model Ecosystem","module":{"index":5,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Grammar-constrained decoding masks illegal tokens at each step so invalid syntax becomes impossible, which converts parse failures into silently wrong values unless the schema gives the model an explicit way to express absence.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["The output parses","The output is true"],"rows":["Prompted for JSON","Grammar-constrained"],"type":"matrix","cells":[["Most of the time","Unchanged"],["Always","Unchanged"]],"title":"What constrained decoding buys, and what it does not","caption":"Constraining converts a parse failure into a confidently wrong value, which is harder to notice. The trade is usually worth making and you have to know you made it: make the field nullable, or add an explicit unknown to the enum, so the model has somewhere to go.","cellLabels":[["a code fence, a preamble, a single quote","it can still invent an amount"],["illegal tokens are masked at every step","a required field with no empty value is an instruction to fabricate"]]},"alt":"Figure: What constrained decoding buys, and what it does not. A two by two grid, rows Prompted for JSON and Grammar-constrained, columns The output parses and The output is true: Prompted for JSON and The output parses: Most of the time; Prompted for JSON and The output is true: Unchanged; Grammar-constrained and The output parses: Always; Grammar-constrained and The output is true: Unchanged. Constraining converts a parse failure into a confidently wrong value, which is harder to notice. The trade is usually worth making and you have to know you made it: make the field nullable, or add an explicit unknown to the enum, so the model has somewhere to go."}},"blocks":[{"t":"h2","id":"asking-is-not-enough","runs":[{"text":"Asking is not enough"}]},{"t":"p","runs":[{"text":"Ask a 7B model to reply with JSON and it will, most of the time. The rest of the time it adds a sentence before the brace, wraps the object in a code fence, uses single quotes, or trails off. Retry loops and regular expressions to pull the object out are the usual response, and they are a bad answer to a problem with a good one."}]},{"t":"p","runs":[{"text":"Constrained decoding removes the failure entirely. At every generation step, a state machine tracks where you are in the grammar, works out which tokens could legally come next, and sets the logits of every other token to negative infinity before sampling. An invalid character is not unlikely; it is impossible."}]},{"t":"h2","id":"using-it","runs":[{"text":"Using it"}]},{"t":"p","runs":[{"text":"llama.cpp","b":true},{"text":", directly from a JSON Schema:"}]},{"t":"code","lang":"bash","label":"Bash","text":"curl localhost:8080/v1/chat/completions -d '{\n  \"messages\":[{\"role\":\"user\",\"content\":\"Extract from: Invoice 8841, 12 March, EUR 4,200, Meridian Ltd\"}],\n  \"temperature\":0,\n  \"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"schema\":{\n    \"type\":\"object\",\n    \"properties\":{\n      \"invoice_no\":{\"type\":\"string\"},\n      \"date\":{\"type\":\"string\",\"format\":\"date\"},\n      \"currency\":{\"type\":\"string\",\"enum\":[\"EUR\",\"USD\",\"GBP\",\"INR\"]},\n      \"amount\":{\"type\":\"number\"},\n      \"supplier\":{\"type\":\"string\"}},\n    \"required\":[\"invoice_no\",\"currency\",\"amount\"],\n    \"additionalProperties\":false}}}}'"},{"t":"p","runs":[{"text":"vLLM","b":true},{"text":" takes "},{"text":"guided_json","code":true},{"text":", "},{"text":"guided_choice","code":true},{"text":" and "},{"text":"guided_regex","code":true},{"text":". "},{"text":"Outlines","b":true},{"text":", "},{"text":"xgrammar","b":true},{"text":" and "},{"text":"lm-format-enforcer","b":true},{"text":" are the libraries underneath most of these, and can be used directly with "},{"text":"transformers","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"llama.cpp also accepts a raw grammar in its own GBNF notation, which is the escape hatch when a schema cannot express what you need:"}]},{"t":"code","lang":"","label":"Text","text":"root   ::= \"SENTIMENT: \" label \"\\nREASON: \" [^\\n]+ \"\\n\"\nlabel  ::= \"positive\" | \"negative\" | \"neutral\""},{"t":"h2","id":"what-it-guarantees-and-what-it-does-not","runs":[{"text":"What it guarantees and what it does not"}]},{"t":"p","runs":[{"text":"It guarantees "},{"text":"syntax","b":true},{"text":". The output will parse, the enum will be one of your values, the number will be a number."}]},{"t":"p","runs":[{"text":"It guarantees nothing about "},{"text":"truth","b":true},{"text":". A model forced to produce an amount will produce an amount, whether or not the document contains one. Constrained decoding converts a parse failure into a confidently wrong value, which is harder to notice. That trade is usually worth making, and you have to know you made it."}]},{"t":"p","runs":[{"text":"Three consequences follow."}]},{"t":"p","runs":[{"text":"Always give the model a way to say nothing.","b":true},{"text":" If a field may be absent, make it nullable or add an explicit "},{"text":"\"unknown\"","code":true},{"text":" to the enum. A required field with no legal empty value is an instruction to fabricate."}]},{"t":"p","runs":[{"text":"Order the fields deliberately.","b":true},{"text":" Generation is left to right, so a field produced early conditions everything after it. Putting a short "},{"text":"evidence","code":true},{"text":" or "},{"text":"reasoning","code":true},{"text":" string before the answer field lets the model work before it commits; putting it after means the reasoning is a justification of a decision already made. This is a real and easily measured effect."}]},{"t":"p","runs":[{"text":"Keep schemas shallow.","b":true},{"text":" Deeply nested objects with many optional branches are harder for small models, and the grammar machinery gets slower to compile. Flat objects with clear names work far better at 7B than an elegant nested design."}]},{"t":"h2","id":"costs","runs":[{"text":"Costs"}]},{"t":"p","runs":[{"text":"Compiling a grammar takes time on first use, so cache it rather than rebuilding per request. Per-token overhead is small in modern implementations. The larger cost is subtler: a heavily constrained format can pull the model off the distribution it writes well in, so quality of the "},{"text":"content","i":true},{"text":" can drop slightly even as validity goes to 100%. Measure both."}]},{"t":"h2","id":"when-not-to-constrain","runs":[{"text":"When not to constrain"}]},{"t":"p","runs":[{"text":"Free-form prose, and any case where you would rather see the model fail visibly than produce a well-formed wrong answer. For a first exploration of a new task, run unconstrained and read what the model naturally produces; it tells you what it understood. Constrain afterwards, once you know the shape you want."}]},{"t":"quote","runs":[{"text":"The practical upgrade path: prompt for JSON, discover the failure rate on 200 real inputs, then constrain and watch it go to zero. Skipping the measurement means you never find out whether the schema you wrote matches the task, because the errors that remain are now silent."}]}],"check":{"question":[{"text":"After enabling JSON Schema constrained decoding, an extraction pipeline’s parse errors drop to zero but downstream data quality gets worse. What is the most likely mechanism?"}],"options":[{"text":[{"text":"Constrained decoding lowers the model’s effective temperature, reducing the diversity needed for accurate extraction."}],"correct":false,"tempting":[{"text":"Masking removes illegal tokens rather than changing the temperature applied to the legal ones."}]},{"text":[{"text":"Required fields with no null or unknown option force the model to emit a value even when the document contains none, so failures become plausible wrong answers instead of errors."}],"correct":true,"tempting":null},{"text":[{"text":"The grammar compilation adds latency, causing timeouts that truncate results."}],"correct":false,"tempting":[{"text":"Compilation is a one-off cost and would produce missing responses, not lower-quality complete ones."}]},{"text":[{"text":"JSON output uses more tokens than free text, so the model runs out of context."}],"correct":false,"tempting":[{"text":"Structured output is usually shorter than prose, and truncation would show as parse failures, which have gone to zero."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"sampling-parameters","title":"The sampler, and the defaults that ship broken","minutes":9},"next":{"slug":"prompting-open-models","title":"What changes when nothing is hidden","minutes":9}}}