{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"structured-output-locally","title":"Output that parses every time","url":"https://addaly.com/learn/running-models-yourself/structured-output-locally","position":35,"of":82,"minutes":9,"answer":"A grammar makes invalid output impossible by zeroing illegal tokens at each step, but it guarantees only shape: put a reasoning field first, keep schemas small.","keyIdea":"A grammar makes invalid output impossible by zeroing illegal tokens at each step, but it guarantees only shape: put a reasoning field first, keep schemas small, and always include a way for the model to say it does not know."},"body":{"format":"markdown","text":"## Asking politely does not scale\n\n\"Respond only with JSON\" works most of the time on a good model, and most of the time is the problem. A small model will occasionally wrap the JSON in prose, add a trailing comma, use single quotes, or begin with a markdown fence. At one failure in fifty, a pipeline processing ten thousand documents produces two hundred exceptions.\n\nLocal inference has a better answer than prompting, and it is one of the genuine advantages of running the model yourself: you control the sampler, so you can make invalid output impossible rather than unlikely.\n\n## Constrained decoding\n\nAt every step the model produces a distribution over the whole vocabulary. A grammar engine computes which tokens could legally come next given the output so far, sets every other token's probability to zero, and samples from what remains.\n\nIf the schema says the next character must be `\"` or `}`, no other token can be chosen. The JSON is valid by construction, not by good behaviour.\n\n```bash\n# llama.cpp with a JSON schema\ncurl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{\n  \"messages\": [{\"role\":\"user\",\"content\":\"Extract name and age.\"}],\n  \"response_format\": {\n    \"type\": \"json_schema\",\n    \"json_schema\": {\"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\"name\": {\"type\":\"string\"}, \"age\": {\"type\":\"integer\"}},\n      \"required\": [\"name\", \"age\"]\n    }}\n  }\n}'\n```\n\nllama.cpp also accepts GBNF grammars directly, which express things a JSON schema cannot:\n\n```\nroot   ::= answer\nanswer ::= \"yes\" | \"no\" | \"unclear\"\n```\n\nThat grammar makes any other output impossible. For classification into a fixed label set, this is the correct tool and it removes an entire category of parsing code.\n\nvLLM and SGLang offer the same through their own backends, and SGLang's implementation is notably fast because it precompiles the schema into a token mask.\n\n> Figure: What a grammar guarantees, and what it does not. Guaranteed by construction: The output parses every time; Only the schema's fields appear; A categorical field holds a permitted value; No markdown fence and no preamble. Still entirely your problem: Whether the extracted number is right; Whether the date exists in the calendar; Whether the model had anything to put there; Whether it should have declined instead. Constrained decoding sets every illegal token to zero probability at each step, so invalid output becomes impossible rather than unlikely. Valid is not correct. And a schema with no way to say the answer is absent has built a machine that fabricates whenever it is.\n\n## The cost nobody mentions\n\nConstrained decoding is not free, and the trade is worth understanding.\n\n**It can reduce answer quality.** Forcing the model down a syntactic path prevents it from reasoning first. A model asked to output `{\"answer\":` immediately has to commit before thinking. The fix is structural: let it produce a short free-text reasoning field *first* in the schema, then the answer. Field order in your schema is generation order, and it matters.\n\n**Over-constraining hurts.** A schema with fifteen required fields and strict enumerations pushes the model into a shape it may not have anything sensible to put in. It will fill the fields anyway, because it must. Valid output is not correct output, and a schema guarantees only the first.\n\n**There is a speed cost**, small in good implementations and noticeable in poor ones, because the legal-token mask has to be computed at every step.\n\n**Tokenisation makes it fiddly.** Grammars operate on characters and models emit tokens that may straddle boundaries, so implementations must handle partial matches. This is why grammar support was slow to arrive and why edge cases still exist with unusual vocabularies.\n\n## Practical design\n\n1. **Keep schemas small.** Three to six fields. Run two passes rather than one enormous schema.\n2. **Put a reasoning field first** if the task needs any judgement.\n3. **Use enumerations for anything categorical.** They eliminate spelling variation and near-synonyms at a stroke.\n4. **Avoid deeply nested objects.** Small models handle flat structures far more reliably.\n5. **Validate anyway.** The grammar guarantees the shape; your code should still check that the age is plausible and the date exists.\n\n## When not to constrain\n\nIf the model needs to say \"I cannot answer this from the given text\", a rigid schema removes its ability to. Include an explicit escape (a `\"status\"` field with an `\"insufficient_information\"` value) or you have built a machine that fabricates whenever the answer is absent, and it will.\n\nThat is the honest limitation of the whole technique. Constrained decoding guarantees the output parses. It says nothing whatever about whether the content is true, and by removing the model's ability to decline, a careless schema makes fabrication more likely rather than less."},"check":{"question":"An extraction pipeline uses a strict JSON schema with required fields for `diagnosis`, `date` and `dosage`. Output always parses, but on documents that mention no dosage the model invents plausible ones. What is the design error?","options":[{"text":"The schema should place a free-text reasoning field before the extracted fields so the model can deliberate first.","correct":false,"tempting":"A genuine improvement for judgement-heavy tasks, but it does not give the model any way to report an absent field, so fabrication would continue."},{"text":"Constrained decoding is unsuitable for extraction because zeroing tokens distorts the probability distribution the model uses to decide what is present.","correct":false,"tempting":"Rejects a technique that works well here; the distortion is over syntax, not over what the document contains."},{"text":"Making every field required removes any way to signal absence, so the grammar forces a value where the correct answer is that there is none.","correct":true},{"text":"The model is too small for medical extraction and a larger one would decline to fill the field.","correct":false,"tempting":"Attributes to capability what the schema makes structurally impossible; no model can emit a token the grammar has zeroed."}]},"prev":{"slug":"the-openai-compatible-api","title":"The API every engine speaks","url":"https://addaly.com/learn/running-models-yourself/the-openai-compatible-api"},"next":{"slug":"embeddings-and-rerankers-locally","title":"Embeddings and rerankers on your own machine","url":"https://addaly.com/learn/running-models-yourself/embeddings-and-rerankers-locally"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Output that parses every time', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/structured-output-locally (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Output that parses every time","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"A grammar makes invalid output impossible by zeroing illegal tokens at each step, but it guarantees only shape: put a reasoning field first, keep schemas small, and always include a way for the model to say it does not know.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Guaranteed by construction","items":["The output parses every time","Only the schema’s fields appear","A categorical field holds a permitted value","No markdown fence and no preamble"]},"type":"compare","right":{"title":"Still entirely your problem","items":["Whether the extracted number is right","Whether the date exists in the calendar","Whether the model had anything to put there","Whether it should have declined instead"]},"title":"What a grammar guarantees, and what it does not","caption":"Constrained decoding sets every illegal token to zero probability at each step, so invalid output becomes impossible rather than unlikely. Valid is not correct. And a schema with no way to say the answer is absent has built a machine that fabricates whenever it is."},"alt":"Figure: What a grammar guarantees, and what it does not. Guaranteed by construction: The output parses every time; Only the schema’s fields appear; A categorical field holds a permitted value; No markdown fence and no preamble. Still entirely your problem: Whether the extracted number is right; Whether the date exists in the calendar; Whether the model had anything to put there; Whether it should have declined instead. Constrained decoding sets every illegal token to zero probability at each step, so invalid output becomes impossible rather than unlikely. Valid is not correct. And a schema with no way to say the answer is absent has built a machine that fabricates whenever it is."}},"blocks":[{"t":"h2","id":"asking-politely-does-not-scale","runs":[{"text":"Asking politely does not scale"}]},{"t":"p","runs":[{"text":"“Respond only with JSON” works most of the time on a good model, and most of the time is the problem. A small model will occasionally wrap the JSON in prose, add a trailing comma, use single quotes, or begin with a markdown fence. At one failure in fifty, a pipeline processing ten thousand documents produces two hundred exceptions."}]},{"t":"p","runs":[{"text":"Local inference has a better answer than prompting, and it is one of the genuine advantages of running the model yourself: you control the sampler, so you can make invalid output impossible rather than unlikely."}]},{"t":"h2","id":"constrained-decoding","runs":[{"text":"Constrained decoding"}]},{"t":"p","runs":[{"text":"At every step the model produces a distribution over the whole vocabulary. A grammar engine computes which tokens could legally come next given the output so far, sets every other token’s probability to zero, and samples from what remains."}]},{"t":"p","runs":[{"text":"If the schema says the next character must be "},{"text":"\"","code":true},{"text":" or "},{"text":"}","code":true},{"text":", no other token can be chosen. The JSON is valid by construction, not by good behaviour."}]},{"t":"code","lang":"bash","label":"Bash","text":"# llama.cpp with a JSON schema\ncurl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{\n  \"messages\": [{\"role\":\"user\",\"content\":\"Extract name and age.\"}],\n  \"response_format\": {\n    \"type\": \"json_schema\",\n    \"json_schema\": {\"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\"name\": {\"type\":\"string\"}, \"age\": {\"type\":\"integer\"}},\n      \"required\": [\"name\", \"age\"]\n    }}\n  }\n}'"},{"t":"p","runs":[{"text":"llama.cpp also accepts GBNF grammars directly, which express things a JSON schema cannot:"}]},{"t":"code","lang":"","label":"Text","text":"root   ::= answer\nanswer ::= \"yes\" | \"no\" | \"unclear\""},{"t":"p","runs":[{"text":"That grammar makes any other output impossible. For classification into a fixed label set, this is the correct tool and it removes an entire category of parsing code."}]},{"t":"p","runs":[{"text":"vLLM and SGLang offer the same through their own backends, and SGLang’s implementation is notably fast because it precompiles the schema into a token mask."}]},{"t":"h2","id":"the-cost-nobody-mentions","runs":[{"text":"The cost nobody mentions"}]},{"t":"p","runs":[{"text":"Constrained decoding is not free, and the trade is worth understanding."}]},{"t":"p","runs":[{"text":"It can reduce answer quality.","b":true},{"text":" Forcing the model down a syntactic path prevents it from reasoning first. A model asked to output "},{"text":"{\"answer\":","code":true},{"text":" immediately has to commit before thinking. The fix is structural: let it produce a short free-text reasoning field "},{"text":"first","i":true},{"text":" in the schema, then the answer. Field order in your schema is generation order, and it matters."}]},{"t":"p","runs":[{"text":"Over-constraining hurts.","b":true},{"text":" A schema with fifteen required fields and strict enumerations pushes the model into a shape it may not have anything sensible to put in. It will fill the fields anyway, because it must. Valid output is not correct output, and a schema guarantees only the first."}]},{"t":"p","runs":[{"text":"There is a speed cost","b":true},{"text":", small in good implementations and noticeable in poor ones, because the legal-token mask has to be computed at every step."}]},{"t":"p","runs":[{"text":"Tokenisation makes it fiddly.","b":true},{"text":" Grammars operate on characters and models emit tokens that may straddle boundaries, so implementations must handle partial matches. This is why grammar support was slow to arrive and why edge cases still exist with unusual vocabularies."}]},{"t":"h2","id":"practical-design","runs":[{"text":"Practical design"}]},{"t":"ol","start":1,"items":[[{"text":"Keep schemas small.","b":true},{"text":" Three to six fields. Run two passes rather than one enormous schema."}],[{"text":"Put a reasoning field first","b":true},{"text":" if the task needs any judgement."}],[{"text":"Use enumerations for anything categorical.","b":true},{"text":" They eliminate spelling variation and near-synonyms at a stroke."}],[{"text":"Avoid deeply nested objects.","b":true},{"text":" Small models handle flat structures far more reliably."}],[{"text":"Validate anyway.","b":true},{"text":" The grammar guarantees the shape; your code should still check that the age is plausible and the date exists."}]]},{"t":"h2","id":"when-not-to-constrain","runs":[{"text":"When not to constrain"}]},{"t":"p","runs":[{"text":"If the model needs to say “I cannot answer this from the given text”, a rigid schema removes its ability to. Include an explicit escape (a "},{"text":"\"status\"","code":true},{"text":" field with an "},{"text":"\"insufficient_information\"","code":true},{"text":" value) or you have built a machine that fabricates whenever the answer is absent, and it will."}]},{"t":"p","runs":[{"text":"That is the honest limitation of the whole technique. Constrained decoding guarantees the output parses. It says nothing whatever about whether the content is true, and by removing the model’s ability to decline, a careless schema makes fabrication more likely rather than less."}]}],"check":{"question":[{"text":"An extraction pipeline uses a strict JSON schema with required fields for "},{"text":"diagnosis","code":true},{"text":", "},{"text":"date","code":true},{"text":" and "},{"text":"dosage","code":true},{"text":". Output always parses, but on documents that mention no dosage the model invents plausible ones. What is the design error?"}],"options":[{"text":[{"text":"The schema should place a free-text reasoning field before the extracted fields so the model can deliberate first."}],"correct":false,"tempting":[{"text":"A genuine improvement for judgement-heavy tasks, but it does not give the model any way to report an absent field, so fabrication would continue."}]},{"text":[{"text":"Constrained decoding is unsuitable for extraction because zeroing tokens distorts the probability distribution the model uses to decide what is present."}],"correct":false,"tempting":[{"text":"Rejects a technique that works well here; the distortion is over syntax, not over what the document contains."}]},{"text":[{"text":"Making every field required removes any way to signal absence, so the grammar forces a value where the correct answer is that there is none."}],"correct":true,"tempting":null},{"text":[{"text":"The model is too small for medical extraction and a larger one would decline to fill the field."}],"correct":false,"tempting":[{"text":"Attributes to capability what the schema makes structurally impossible; no model can emit a token the grammar has zeroed."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"the-openai-compatible-api","title":"The API every engine speaks","minutes":8},"next":{"slug":"embeddings-and-rerankers-locally","title":"Embeddings and rerankers on your own machine","minutes":10}}}