{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"how-a-schema-is-enforced","title":"How a schema is actually enforced","url":"https://addaly.com/learn/context-engineering/how-a-schema-is-enforced","position":59,"of":87,"minutes":10,"answer":"Structured output is implemented by masking the model's logits at each step so only tokens that continue a valid parse survive, which guarantees a document that parses and guarantees nothing about whether it is true.","keyIdea":"Structured output is implemented by masking the model's logits at each step so only tokens that continue a valid parse survive, which guarantees a document that parses and guarantees nothing about whether it is true, complete, or an honest answer to the question."},"body":{"format":"markdown","text":"## Masking, not asking\n\nWhen you set a response format and get back valid JSON every time, no part of that reliability came from the model deciding to cooperate. It came from arithmetic performed after the model spoke and before the token was chosen.\n\nAt every decoding step the model produces a logit for each token in its vocabulary: perhaps 130,000 numbers. Ordinarily the sampler picks from that distribution. Under constrained decoding, a separate component first computes which tokens could legally continue the output given the grammar and what has been emitted so far, and sets the logit of every other token to negative infinity. After the softmax, those tokens have probability zero. They cannot be sampled. Not unlikely: impossible.\n\nIf the schema says the next thing must be one of three enum strings and the model has already emitted `\"status\": \"`, then the only permitted next tokens are those that begin one of those three strings. The model's preference among the three still decides the outcome. Its preference for a fourth word it invented does not get a vote.\n\nThe machinery that computes the legal set is a state machine compiled from the schema: a finite automaton for regular structures, a pushdown automaton where nesting has to be tracked. You can read the implementations: `outlines` and `xgrammar` in Python, GBNF grammars in `llama.cpp`. All free, all local, all runnable on a laptop, and reading one for twenty minutes is worth more than any amount of speculation about what \"JSON mode\" does.\n\n> Figure: One decoding step under a schema. Steps: 1. A logit for every token (Perhaps 130,000 numbers, exactly as there would be without a schema); 2. An automaton reads the output so far (Compiled from the schema, with a stack wherever nesting has to be tracked); 3. Illegal continuations are masked (Anything that cannot continue a valid parse is set to minus infinity); 4. The sampler picks what is left (After the model spoke and before the token was chosen); 5. The document parses, every time (Required fields present, enums in range, integers made of digits); 6. And none of it is true yet (The integer can be wrong and the citation can name no passage at all). None of the reliability came from the model deciding to cooperate. The most common way this makes a system quietly worse is a closed enum with no escape: a customer asking what time you close is classified as a refund, with no hedging, because the format gives it nowhere to hedge.\n\n## What this buys, precisely\n\nIt buys syntax. The output will parse. Required fields will be present. An enum field will hold one of the enum values. A field typed as integer will contain digits.\n\nThat is a genuine and large win. Before constrained decoding, the standard production pattern was: call, try to parse, fail on roughly one call in fifty, retry with the error appended, and carry the retry cost and latency forever. That failure mode is gone.\n\nIt is also the entire list. Consider what the guarantee does not touch:\n\n- The integer can be wrong.\n- The date can parse as a date and describe a day the event did not happen.\n- The `citations` array can be present, correctly typed, and contain identifiers for passages that were never in the window.\n- A field typed as string can contain an apology.\n\nThe schema is a constraint on the shape of the container. Nothing about the container constrains its contents.\n\n## The enum trap\n\nThis one is worth its own heading because it is the most common way a schema makes a system quietly worse.\n\nSuppose you classify support messages into `refund`, `replacement`, `escalate`. The schema constrains the output to those three. Now a customer writes to ask what time you close.\n\nThe model cannot say \"none of these\". The tokens for that answer have probability zero. It will emit one of the three, and it will emit it with no hedging, because the format gives it nowhere to hedge. Downstream, your code reads `escalate` and creates a ticket. The confidence you perceive in that output is an artefact of the mask.\n\nThe fix is in the schema, not the prompt:\n\n```json\n\"intent\": {\n  \"type\": \"string\",\n  \"enum\": [\"refund\", \"replacement\", \"escalate\", \"other\", \"unclear\"]\n}\n```\n\n`other` for messages that are about something else, `unclear` for messages too ambiguous to classify. Both routed to a human. Then measure how often each fires. If `other` never fires on real traffic, your enum is probably fine or your model is ignoring the escape hatch, and you can tell which by sampling. If it fires on eight percent, you have just discovered eight percent of your traffic that used to be silently misrouted.\n\nEvery closed set of options needs an explicit escape. This applies to boolean fields too: a required boolean forces a claim where the honest answer is that the document does not say.\n\n## The costs, which are small but real\n\n**Compilation.** The automaton is built from the schema. For a simple flat object this is a millisecond. For deeply nested schemas with many optional branches and regular-expression constraints it can reach hundreds of milliseconds. It is cacheable per schema, and every serious implementation caches it; but a system that generates a fresh schema per request defeats that cache and pays the build every time.\n\n**Per-token overhead.** Computing and applying the mask costs time at each step. Modern implementations do it concurrently with the forward pass and the overhead is usually under a few percent. It is not zero.\n\n**Quality, and this is the one to watch.** Forcing structure from the first token removes the model's room to work. If the schema's first field is `answer`, the model must commit before it has reasoned. On tasks that need any intermediate steps, this measurably degrades accuracy.\n\nThe remedy is to give the reasoning a field of its own and put it first:\n\n```json\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"evidence\": {\"type\": \"string\", \"description\": \"Quote the passage this rests on.\"},\n    \"reasoning\": {\"type\": \"string\"},\n    \"answer\": {\"type\": \"string\"}\n  },\n  \"required\": [\"evidence\", \"reasoning\", \"answer\"]\n}\n```\n\nJSON object fields are generated in schema order. Put the thinking before the conclusion and the conclusion is conditioned on it. Put it after and it is a justification written for a decision already made, which is a different and much less useful thing, and reads identically.\n\n## Where the ground is not settled\n\nWhether constrained decoding degrades reasoning even when the field order is sensible is still argued. Some studies report a real drop on reasoning benchmarks under strict schemas; others attribute most of the gap to prompt and ordering differences rather than to masking itself. The honest position is that the effect exists, its size depends on model and task, and it is small enough that you will not notice it without measuring.\n\nSo measure it. Run your golden set free-form with the schema described in prose, and again under enforcement, and compare accuracy on the field that matters. If enforcement costs you two points of accuracy and saves a retry path, that is a trade you can now actually make, instead of guessing."},"check":{"question":"A classifier is constrained to the enum [refund, replacement, escalate]. A user asks about opening hours. What does the constraint cause?","options":[{"text":"The model returns an empty string, since none of the values apply and the field is a string type.","correct":false,"tempting":"An empty string is not a member of the enum, so the mask forbids it as firmly as any invented word."},{"text":"The model emits a low-confidence signal alongside the chosen value, which the code can threshold on.","correct":false,"tempting":"Assumes a calibrated confidence exists in the response; the constraint suppresses exactly the hedging that would have shown uncertainty."},{"text":"The request fails validation and is retried, because no enum member fits the input.","correct":false,"tempting":"Validation failure is what constrained decoding prevents; the output is valid by construction, which is why the failure is silent."},{"text":"One of the three is emitted confidently, since every other token has probability zero.","correct":true}]},"prev":{"slug":"forcing-a-schema","title":"Structured output, and what a schema does not buy you","url":"https://addaly.com/learn/context-engineering/forcing-a-schema"},"next":{"slug":"tool-definitions-are-context","title":"Every tool costs tokens before anyone calls it","url":"https://addaly.com/learn/context-engineering/tool-definitions-are-context"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How a schema is actually enforced', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/how-a-schema-is-enforced (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How a schema is actually enforced","courseTitle":"Context Engineering","module":{"index":7,"slug":"tools-schemas-and-the-agent-window","title":"Tools, schemas and the agent’s window"},"answer":null,"keyIdea":"Structured output is implemented by masking the model’s logits at each step so only tokens that continue a valid parse survive, which guarantees a document that parses and guarantees nothing about whether it is true, complete, or an honest answer to the question.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"A logit for every token","detail":"Perhaps 130,000 numbers, exactly as there would be without a schema"},{"label":"An automaton reads the output so far","detail":"Compiled from the schema, with a stack wherever nesting has to be tracked"},{"label":"Illegal continuations are masked","detail":"Anything that cannot continue a valid parse is set to minus infinity"},{"label":"The sampler picks what is left","detail":"After the model spoke and before the token was chosen"},{"label":"The document parses, every time","detail":"Required fields present, enums in range, integers made of digits"},{"label":"And none of it is true yet","detail":"The integer can be wrong and the citation can name no passage at all"}],"title":"One decoding step under a schema","caption":"None of the reliability came from the model deciding to cooperate. The most common way this makes a system quietly worse is a closed enum with no escape: a customer asking what time you close is classified as a refund, with no hedging, because the format gives it nowhere to hedge."},"alt":"Figure: One decoding step under a schema. Steps: 1. A logit for every token (Perhaps 130,000 numbers, exactly as there would be without a schema); 2. An automaton reads the output so far (Compiled from the schema, with a stack wherever nesting has to be tracked); 3. Illegal continuations are masked (Anything that cannot continue a valid parse is set to minus infinity); 4. The sampler picks what is left (After the model spoke and before the token was chosen); 5. The document parses, every time (Required fields present, enums in range, integers made of digits); 6. And none of it is true yet (The integer can be wrong and the citation can name no passage at all). None of the reliability came from the model deciding to cooperate. The most common way this makes a system quietly worse is a closed enum with no escape: a customer asking what time you close is classified as a refund, with no hedging, because the format gives it nowhere to hedge."}},"blocks":[{"t":"h2","id":"masking-not-asking","runs":[{"text":"Masking, not asking"}]},{"t":"p","runs":[{"text":"When you set a response format and get back valid JSON every time, no part of that reliability came from the model deciding to cooperate. It came from arithmetic performed after the model spoke and before the token was chosen."}]},{"t":"p","runs":[{"text":"At every decoding step the model produces a logit for each token in its vocabulary: perhaps 130,000 numbers. Ordinarily the sampler picks from that distribution. Under constrained decoding, a separate component first computes which tokens could legally continue the output given the grammar and what has been emitted so far, and sets the logit of every other token to negative infinity. After the softmax, those tokens have probability zero. They cannot be sampled. Not unlikely: impossible."}]},{"t":"p","runs":[{"text":"If the schema says the next thing must be one of three enum strings and the model has already emitted "},{"text":"\"status\": \"","code":true},{"text":", then the only permitted next tokens are those that begin one of those three strings. The model’s preference among the three still decides the outcome. Its preference for a fourth word it invented does not get a vote."}]},{"t":"p","runs":[{"text":"The machinery that computes the legal set is a state machine compiled from the schema: a finite automaton for regular structures, a pushdown automaton where nesting has to be tracked. You can read the implementations: "},{"text":"outlines","code":true},{"text":" and "},{"text":"xgrammar","code":true},{"text":" in Python, GBNF grammars in "},{"text":"llama.cpp","code":true},{"text":". All free, all local, all runnable on a laptop, and reading one for twenty minutes is worth more than any amount of speculation about what “JSON mode” does."}]},{"t":"h2","id":"what-this-buys-precisely","runs":[{"text":"What this buys, precisely"}]},{"t":"p","runs":[{"text":"It buys syntax. The output will parse. Required fields will be present. An enum field will hold one of the enum values. A field typed as integer will contain digits."}]},{"t":"p","runs":[{"text":"That is a genuine and large win. Before constrained decoding, the standard production pattern was: call, try to parse, fail on roughly one call in fifty, retry with the error appended, and carry the retry cost and latency forever. That failure mode is gone."}]},{"t":"p","runs":[{"text":"It is also the entire list. Consider what the guarantee does not touch:"}]},{"t":"ul","items":[[{"text":"The integer can be wrong."}],[{"text":"The date can parse as a date and describe a day the event did not happen."}],[{"text":"The "},{"text":"citations","code":true},{"text":" array can be present, correctly typed, and contain identifiers for passages that were never in the window."}],[{"text":"A field typed as string can contain an apology."}]]},{"t":"p","runs":[{"text":"The schema is a constraint on the shape of the container. Nothing about the container constrains its contents."}]},{"t":"h2","id":"the-enum-trap","runs":[{"text":"The enum trap"}]},{"t":"p","runs":[{"text":"This one is worth its own heading because it is the most common way a schema makes a system quietly worse."}]},{"t":"p","runs":[{"text":"Suppose you classify support messages into "},{"text":"refund","code":true},{"text":", "},{"text":"replacement","code":true},{"text":", "},{"text":"escalate","code":true},{"text":". The schema constrains the output to those three. Now a customer writes to ask what time you close."}]},{"t":"p","runs":[{"text":"The model cannot say “none of these”. The tokens for that answer have probability zero. It will emit one of the three, and it will emit it with no hedging, because the format gives it nowhere to hedge. Downstream, your code reads "},{"text":"escalate","code":true},{"text":" and creates a ticket. The confidence you perceive in that output is an artefact of the mask."}]},{"t":"p","runs":[{"text":"The fix is in the schema, not the prompt:"}]},{"t":"code","lang":"json","label":"JSON","text":"\"intent\": {\n  \"type\": \"string\",\n  \"enum\": [\"refund\", \"replacement\", \"escalate\", \"other\", \"unclear\"]\n}"},{"t":"p","runs":[{"text":"other","code":true},{"text":" for messages that are about something else, "},{"text":"unclear","code":true},{"text":" for messages too ambiguous to classify. Both routed to a human. Then measure how often each fires. If "},{"text":"other","code":true},{"text":" never fires on real traffic, your enum is probably fine or your model is ignoring the escape hatch, and you can tell which by sampling. If it fires on eight percent, you have just discovered eight percent of your traffic that used to be silently misrouted."}]},{"t":"p","runs":[{"text":"Every closed set of options needs an explicit escape. This applies to boolean fields too: a required boolean forces a claim where the honest answer is that the document does not say."}]},{"t":"h2","id":"the-costs-which-are-small-but-real","runs":[{"text":"The costs, which are small but real"}]},{"t":"p","runs":[{"text":"Compilation.","b":true},{"text":" The automaton is built from the schema. For a simple flat object this is a millisecond. For deeply nested schemas with many optional branches and regular-expression constraints it can reach hundreds of milliseconds. It is cacheable per schema, and every serious implementation caches it; but a system that generates a fresh schema per request defeats that cache and pays the build every time."}]},{"t":"p","runs":[{"text":"Per-token overhead.","b":true},{"text":" Computing and applying the mask costs time at each step. Modern implementations do it concurrently with the forward pass and the overhead is usually under a few percent. It is not zero."}]},{"t":"p","runs":[{"text":"Quality, and this is the one to watch.","b":true},{"text":" Forcing structure from the first token removes the model’s room to work. If the schema’s first field is "},{"text":"answer","code":true},{"text":", the model must commit before it has reasoned. On tasks that need any intermediate steps, this measurably degrades accuracy."}]},{"t":"p","runs":[{"text":"The remedy is to give the reasoning a field of its own and put it first:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\n  \"type\": \"object\",\n  \"properties\": {\n    \"evidence\": {\"type\": \"string\", \"description\": \"Quote the passage this rests on.\"},\n    \"reasoning\": {\"type\": \"string\"},\n    \"answer\": {\"type\": \"string\"}\n  },\n  \"required\": [\"evidence\", \"reasoning\", \"answer\"]\n}"},{"t":"p","runs":[{"text":"JSON object fields are generated in schema order. Put the thinking before the conclusion and the conclusion is conditioned on it. Put it after and it is a justification written for a decision already made, which is a different and much less useful thing, and reads identically."}]},{"t":"h2","id":"where-the-ground-is-not-settled","runs":[{"text":"Where the ground is not settled"}]},{"t":"p","runs":[{"text":"Whether constrained decoding degrades reasoning even when the field order is sensible is still argued. Some studies report a real drop on reasoning benchmarks under strict schemas; others attribute most of the gap to prompt and ordering differences rather than to masking itself. The honest position is that the effect exists, its size depends on model and task, and it is small enough that you will not notice it without measuring."}]},{"t":"p","runs":[{"text":"So measure it. Run your golden set free-form with the schema described in prose, and again under enforcement, and compare accuracy on the field that matters. If enforcement costs you two points of accuracy and saves a retry path, that is a trade you can now actually make, instead of guessing."}]}],"check":{"question":[{"text":"A classifier is constrained to the enum [refund, replacement, escalate]. A user asks about opening hours. What does the constraint cause?"}],"options":[{"text":[{"text":"The model returns an empty string, since none of the values apply and the field is a string type."}],"correct":false,"tempting":[{"text":"An empty string is not a member of the enum, so the mask forbids it as firmly as any invented word."}]},{"text":[{"text":"The model emits a low-confidence signal alongside the chosen value, which the code can threshold on."}],"correct":false,"tempting":[{"text":"Assumes a calibrated confidence exists in the response; the constraint suppresses exactly the hedging that would have shown uncertainty."}]},{"text":[{"text":"The request fails validation and is retried, because no enum member fits the input."}],"correct":false,"tempting":[{"text":"Validation failure is what constrained decoding prevents; the output is valid by construction, which is why the failure is silent."}]},{"text":[{"text":"One of the three is emitted confidently, since every other token has probability zero."}],"correct":true,"tempting":null}],"order":[0,3,1,2],"why":null},"prev":{"slug":"forcing-a-schema","title":"Structured output, and what a schema does not buy you","minutes":9},"next":{"slug":"tool-definitions-are-context","title":"Every tool costs tokens before anyone calls it","minutes":9}}}