{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"constrained-decoding","title":"Making output structurally valid, without asking nicely","url":"https://addaly.com/learn/how-llms-work/constrained-decoding","position":25,"of":78,"minutes":9,"answer":"Structure can be enforced by masking impossible tokens at each step, which makes invalid output impossible rather than unlikely.","keyIdea":"Structure can be enforced by masking impossible tokens at each step, which makes invalid output impossible rather than unlikely."},"body":{"format":"markdown","text":"## The wrong way, which everybody tries first\n\n\"Respond only with valid JSON. Do not include any explanation.\" Then a retry loop, then a regular expression to strip the markdown fence the model added anyway, then a bug report about the one time in five hundred that a string contained an unescaped quotation mark.\n\nPrompting for structure makes valid output *likely*. It cannot make it *certain*, because the model is producing a distribution over every token in its vocabulary at every step and nothing forbids the wrong one.\n\n## The right way, which is mechanical\n\nAt each step you have logits for all 128,256 tokens. You also know, from the partial output so far, which tokens could possibly come next in a valid structure. So set the logits of all the others to negative infinity before sampling.\n\n```python\nallowed = grammar.allowed_next_tokens(generated_so_far)   # a set of token ids\nlogits[~allowed] = -float(\"inf\")\nnext_id = sample(softmax(logits))\n```\n\nInvalid output is now impossible, not improbable. If the current state is \"inside a JSON object, expecting a key\", the only permitted tokens are those that begin a quoted string. If a schema says the `status` field must be one of three enum values, then after `\"status\": \"` only the tokens that begin those three values survive.\n\nThis is variously called constrained decoding, grammar-constrained generation, or structured output, and every major provider now offers a version of it.\n\n## What you can use, free\n\n- **`llama.cpp`** accepts a GBNF grammar file directly. Fully offline, runs on a laptop.\n- **Outlines**, **LMQL** and **XGrammar** are open-source Python libraries that compile a JSON Schema, a regular expression or a context-free grammar into a token mask, and work with local models and several APIs.\n- Hosted providers expose it as a structured-output or JSON-schema mode. The implementation is the same masking; you supply a schema instead of a grammar.\n\nA JSON Schema is the usual entry point because most people already have one, and the library does the compilation.\n\n## The honest limitations\n\nFour, and they matter.\n\n**Valid is not correct.** A constrained model will produce a perfectly well-formed object with a fabricated invoice number. You have removed a class of parsing failures, not a class of factual ones.\n\n**Constraint can degrade content.** Forcing structure at every step can push the model off a path it would otherwise have taken. Published results disagree about the size of this effect and it appears to depend on the schema and the task: a tight schema with unnatural field names costs more than a loose one. Where quality matters, measure the constrained and unconstrained versions on your own set rather than assuming the constraint is free.\n\n**Field order is not free.** The model generates left to right, so a field that appears before the fields it depends on has to be produced without them. Putting `\"answer\"` before `\"reasoning\"` in your schema removes the model's chance to work through the problem in the output, which module three's first lesson explained is where visible reasoning happens. Order your schema so the derived fields come last.\n\n**Compilation costs something.** Building the mask machinery for a complex grammar takes time and memory, though modern implementations cache it and the per-token overhead is small.\n\n## Where it goes beyond JSON\n\nThe same mechanism enforces anything expressible as a grammar: a date format, a SQL dialect, a restricted set of function names, an output that must be one of exactly four labels, a chess move in algebraic notation. The classification trick from the previous lesson is a one-token special case of it.\n\nFor a tool-calling agent this matters more than it first appears. Constraining generated calls to your actual function signatures eliminates the entire category of \"the model invented a parameter\", which otherwise has to be caught by validation and retried.\n\n## The decision rule\n\nUse constrained decoding whenever the output is consumed by code. Do not use it for prose. And keep in mind what it is: a filter over the distribution the model produced, applied token by token. It changes what can be emitted, never what the model knows."},"check":{"question":"A schema is defined with fields in the order `verdict`, then `evidence`, then `reasoning`. Constrained decoding guarantees valid output, but the verdicts are noticeably worse than the same model produced in free prose. Why?","options":[{"text":"The constraint mask distorts the probability distribution and degrades all output equally.","correct":false,"tempting":"Constraint really can degrade quality, so a general distortion story is close enough to sound right without identifying the specific cause."},{"text":"The verdict is generated first, so the model has no opportunity to work through the evidence in its output before committing to it.","correct":true},{"text":"Field names in the schema are unfamiliar tokens that push the model off distribution.","correct":false,"tempting":"Unnatural field names do cost something, which makes this a real effect proposed at the wrong scale."},{"text":"JSON output uses more tokens, leaving less context for the reasoning.","correct":false,"tempting":"Format overhead is genuine and context pressure is a real constraint, so combining them sounds like careful engineering."}]},"prev":{"slug":"stop-tokens-and-chat-templates","title":"Special tokens, stop conditions, and where a turn ends","url":"https://addaly.com/learn/how-llms-work/stop-tokens-and-chat-templates"},"next":{"slug":"sampling-many-answers","title":"Asking the same question several times","url":"https://addaly.com/learn/how-llms-work/sampling-many-answers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Making output structurally valid, without asking nicely', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/constrained-decoding (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Making output structurally valid, without asking nicely","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Structure can be enforced by masking impossible tokens at each step, which makes invalid output impossible rather than unlikely.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-wrong-way-which-everybody-tries-first","runs":[{"text":"The wrong way, which everybody tries first"}]},{"t":"p","runs":[{"text":"“Respond only with valid JSON. Do not include any explanation.” Then a retry loop, then a regular expression to strip the markdown fence the model added anyway, then a bug report about the one time in five hundred that a string contained an unescaped quotation mark."}]},{"t":"p","runs":[{"text":"Prompting for structure makes valid output "},{"text":"likely","i":true},{"text":". It cannot make it "},{"text":"certain","i":true},{"text":", because the model is producing a distribution over every token in its vocabulary at every step and nothing forbids the wrong one."}]},{"t":"h2","id":"the-right-way-which-is-mechanical","runs":[{"text":"The right way, which is mechanical"}]},{"t":"p","runs":[{"text":"At each step you have logits for all 128,256 tokens. You also know, from the partial output so far, which tokens could possibly come next in a valid structure. So set the logits of all the others to negative infinity before sampling."}]},{"t":"code","lang":"python","label":"Python","text":"allowed = grammar.allowed_next_tokens(generated_so_far)   # a set of token ids\nlogits[~allowed] = -float(\"inf\")\nnext_id = sample(softmax(logits))"},{"t":"p","runs":[{"text":"Invalid output is now impossible, not improbable. If the current state is “inside a JSON object, expecting a key”, the only permitted tokens are those that begin a quoted string. If a schema says the "},{"text":"status","code":true},{"text":" field must be one of three enum values, then after "},{"text":"\"status\": \"","code":true},{"text":" only the tokens that begin those three values survive."}]},{"t":"p","runs":[{"text":"This is variously called constrained decoding, grammar-constrained generation, or structured output, and every major provider now offers a version of it."}]},{"t":"h2","id":"what-you-can-use-free","runs":[{"text":"What you can use, free"}]},{"t":"ul","items":[[{"text":"llama.cpp","b":true,"code":true},{"text":" accepts a GBNF grammar file directly. Fully offline, runs on a laptop."}],[{"text":"Outlines","b":true},{"text":", "},{"text":"LMQL","b":true},{"text":" and "},{"text":"XGrammar","b":true},{"text":" are open-source Python libraries that compile a JSON Schema, a regular expression or a context-free grammar into a token mask, and work with local models and several APIs."}],[{"text":"Hosted providers expose it as a structured-output or JSON-schema mode. The implementation is the same masking; you supply a schema instead of a grammar."}]]},{"t":"p","runs":[{"text":"A JSON Schema is the usual entry point because most people already have one, and the library does the compilation."}]},{"t":"h2","id":"the-honest-limitations","runs":[{"text":"The honest limitations"}],"role":"limit"},{"t":"p","runs":[{"text":"Four, and they matter."}]},{"t":"p","runs":[{"text":"Valid is not correct.","b":true},{"text":" A constrained model will produce a perfectly well-formed object with a fabricated invoice number. You have removed a class of parsing failures, not a class of factual ones."}]},{"t":"p","runs":[{"text":"Constraint can degrade content.","b":true},{"text":" Forcing structure at every step can push the model off a path it would otherwise have taken. Published results disagree about the size of this effect and it appears to depend on the schema and the task: a tight schema with unnatural field names costs more than a loose one. Where quality matters, measure the constrained and unconstrained versions on your own set rather than assuming the constraint is free."}]},{"t":"p","runs":[{"text":"Field order is not free.","b":true},{"text":" The model generates left to right, so a field that appears before the fields it depends on has to be produced without them. Putting "},{"text":"\"answer\"","code":true},{"text":" before "},{"text":"\"reasoning\"","code":true},{"text":" in your schema removes the model’s chance to work through the problem in the output, which module three’s first lesson explained is where visible reasoning happens. Order your schema so the derived fields come last."}]},{"t":"p","runs":[{"text":"Compilation costs something.","b":true},{"text":" Building the mask machinery for a complex grammar takes time and memory, though modern implementations cache it and the per-token overhead is small."}]},{"t":"h2","id":"where-it-goes-beyond-json","runs":[{"text":"Where it goes beyond JSON"}]},{"t":"p","runs":[{"text":"The same mechanism enforces anything expressible as a grammar: a date format, a SQL dialect, a restricted set of function names, an output that must be one of exactly four labels, a chess move in algebraic notation. The classification trick from the previous lesson is a one-token special case of it."}]},{"t":"p","runs":[{"text":"For a tool-calling agent this matters more than it first appears. Constraining generated calls to your actual function signatures eliminates the entire category of “the model invented a parameter”, which otherwise has to be caught by validation and retried."}]},{"t":"h2","id":"the-decision-rule","runs":[{"text":"The decision rule"}]},{"t":"p","runs":[{"text":"Use constrained decoding whenever the output is consumed by code. Do not use it for prose. And keep in mind what it is: a filter over the distribution the model produced, applied token by token. It changes what can be emitted, never what the model knows."}]}],"check":{"question":[{"text":"A schema is defined with fields in the order "},{"text":"verdict","code":true},{"text":", then "},{"text":"evidence","code":true},{"text":", then "},{"text":"reasoning","code":true},{"text":". Constrained decoding guarantees valid output, but the verdicts are noticeably worse than the same model produced in free prose. Why?"}],"options":[{"text":[{"text":"The constraint mask distorts the probability distribution and degrades all output equally."}],"correct":false,"tempting":[{"text":"Constraint really can degrade quality, so a general distortion story is close enough to sound right without identifying the specific cause."}]},{"text":[{"text":"The verdict is generated first, so the model has no opportunity to work through the evidence in its output before committing to it."}],"correct":true,"tempting":null},{"text":[{"text":"Field names in the schema are unfamiliar tokens that push the model off distribution."}],"correct":false,"tempting":[{"text":"Unnatural field names do cost something, which makes this a real effect proposed at the wrong scale."}]},{"text":[{"text":"JSON output uses more tokens, leaving less context for the reasoning."}],"correct":false,"tempting":[{"text":"Format overhead is genuine and context pressure is a real constraint, so combining them sounds like careful engineering."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"stop-tokens-and-chat-templates","title":"Special tokens, stop conditions, and where a turn ends","minutes":8},"next":{"slug":"sampling-many-answers","title":"Asking the same question several times","minutes":8}}}