{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"structure-without-training","title":"Getting structure without training for it","url":"https://addaly.com/learn/fine-tuning/structure-without-training","position":64,"of":84,"minutes":9,"answer":"Constrained decoding masks illegal tokens at every step so invalid output cannot be produced, which makes it a free floor under any structured task: it guarantees syntax and nothing about truth.","keyIdea":"Constrained decoding masks illegal tokens at every step so invalid output cannot be produced, which makes it a free floor under any structured task: it guarantees syntax and nothing about truth, so keep it even after fine-tuning rather than treating the two as alternatives."},"body":{"format":"markdown","text":"## Make invalid output impossible\n\nFine-tuning for format makes bad output *rarer*. Constrained decoding makes it *impossible*, and it needs no training at all.\n\nThe mechanism is simple and worth understanding rather than treating as magic. At every generation step the model produces a logit for each of 128,000 tokens. A constrained decoder maintains a state machine built from your schema, works out which tokens could legally come next given what has been generated, and sets every other logit to negative infinity before sampling.\n\nIf the schema says the next character must be `\"` or `}`, then every token that does not begin with one of those has zero probability. The model cannot produce invalid JSON, not because it has learned not to, but because the invalid continuations were removed from the choice.\n\n## The tools, all free\n\n```python\nfrom outlines import models, generate\nfrom pydantic import BaseModel\n\nclass Triage(BaseModel):\n    queue: Literal[\"billing\",\"shipping\",\"refunds\",\"technical\",\"other\"]\n    urgency: int\n    summary: str\n\nresult = generate.json(models.transformers(\"Qwen/Qwen2.5-7B-Instruct\"), Triage)(ticket)\n```\n\nThat returns a validated object, always. Alternatives: `llama.cpp` supports GBNF grammars natively; vLLM has guided decoding built in; XGrammar and lm-format-enforcer provide the same capability with different performance characteristics. Nothing here costs money.\n\nGrammars are more general than JSON schemas. Anything you can express as a context-free grammar (a SQL subset, a domain-specific language, a fixed report template, a limited set of tool calls) can be enforced the same way.\n\n## What it guarantees, and what it does not\n\n**It guarantees syntax.** The output parses, the enum value is one of your five, the integer is an integer, the required fields are present.\n\n**It guarantees nothing about truth.** A constrained decoder will happily emit `{\"queue\": \"billing\", \"urgency\": 3}` for a shipping complaint. Validity and correctness are different properties, and confusing them is the commonest mistake here.\n\nThere is also a real debate about whether heavy constraints reduce answer quality. The argument for: forcing a model into a rigid shape from the first token prevents it producing the intermediate reasoning it would otherwise use. The argument against: careful comparisons find little or no degradation when the schema is reasonable and the model is competent. Both positions have supporting experiments, the effect appears to depend heavily on the task and the schema, and it is not settled.\n\nThe pragmatic response is a design that sidesteps it: let the model reason freely in a `reasoning` string field that comes *first* in the schema, then constrain the structured fields that follow. You get the thinking and the guarantee.\n\n## The poor version, for hosted APIs\n\nIf you are calling an API that offers no grammar support, validate and retry:\n\n```python\nfor attempt in range(3):\n    out = call_model(prompt)\n    try:\n        return Triage.model_validate_json(out)\n    except ValidationError as e:\n        prompt += f\"\\nThat was invalid: {e}. Return only valid JSON.\"\nraise\n```\n\nThis works, and it costs a full extra generation every time it fires. At a 4% invalid rate on a million requests that is 40,000 extra calls. Constrained decoding costs nothing per request; retries cost a generation per failure. That arithmetic is how you decide.\n\nMany hosted providers now offer a structured-output mode that does the constraining server-side. Use it where it exists.\n\n## When fine-tuning for format is still right\n\nThree cases survive:\n\n1. **The schema is enormous.** A grammar with hundreds of fields makes the state machine large and the per-token masking expensive. A model that has learned the shape needs less help from the constraint.\n2. **The model must choose the schema.** Constraining requires knowing which grammar to apply. If deciding *which* tool to call, or which of twelve document types this is, is itself the task, that decision cannot be constrained, though it can often be made by a cheap classifier, as module one argued.\n3. **The content inside the structure is wrong.** If the JSON is valid and the values are poor, no constraint helps. That is a knowledge or capability problem, diagnosed as such.\n\n## The order to try things\n\n1. Constrained decoding. Free, immediate, guaranteed.\n2. Constrained decoding plus a few examples in the prompt, which improves the *content* inside the valid structure.\n3. Fine-tuning, if the content is still wrong after both, and then keep the constraint as well, because a fine-tuned model still produces invalid output occasionally and the constraint costs nothing to keep.\n\nThat last point is the one to remember. These are not alternatives. Constrained decoding is a floor you should never remove, and fine-tuning raises what happens above it."},"check":{"question":"A team adds JSON-schema constrained decoding and their valid-output rate goes from 91% to 100%, but routing accuracy is unchanged. What does this show?","options":[{"text":"The constraint is too permissive; tightening the schema with stricter field types would improve accuracy too.","correct":false,"tempting":"Assumes accuracy is reachable through the schema, when a stricter schema still cannot distinguish a correct queue from a plausible one."},{"text":"The constraint fixed syntax, which is not what the accuracy metric measured.","correct":true},{"text":"The 9% of outputs that were invalid were also being counted as routing errors, so accuracy should have risen by 9 points.","correct":false,"tempting":"Assumes the invalid outputs were scored as wrong answers, which would make the unchanged accuracy a contradiction rather than an observation."},{"text":"Constrained decoding was masking the model's reasoning tokens, which cancelled out the gain from valid formatting.","correct":false,"tempting":"Invokes the contested quality-degradation argument to explain a flat result, when the accuracy was simply unaffected in either direction."}]},"prev":{"slug":"diffusion-lora-and-likeness","title":"LoRA for image models, and the likeness question","url":"https://addaly.com/learn/fine-tuning/diffusion-lora-and-likeness"},"next":{"slug":"small-models-and-edge","title":"Small models, where fine-tuning matters most","url":"https://addaly.com/learn/fine-tuning/small-models-and-edge"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Getting structure without training for it', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/structure-without-training (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Getting structure without training for it","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"Constrained decoding masks illegal tokens at every step so invalid output cannot be produced, which makes it a free floor under any structured task: it guarantees syntax and nothing about truth, so keep it even after fine-tuning rather than treating the two as alternatives.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"make-invalid-output-impossible","runs":[{"text":"Make invalid output impossible"}]},{"t":"p","runs":[{"text":"Fine-tuning for format makes bad output "},{"text":"rarer","i":true},{"text":". Constrained decoding makes it "},{"text":"impossible","i":true},{"text":", and it needs no training at all."}]},{"t":"p","runs":[{"text":"The mechanism is simple and worth understanding rather than treating as magic. At every generation step the model produces a logit for each of 128,000 tokens. A constrained decoder maintains a state machine built from your schema, works out which tokens could legally come next given what has been generated, and sets every other logit to negative infinity before sampling."}]},{"t":"p","runs":[{"text":"If the schema says the next character must be "},{"text":"\"","code":true},{"text":" or "},{"text":"}","code":true},{"text":", then every token that does not begin with one of those has zero probability. The model cannot produce invalid JSON, not because it has learned not to, but because the invalid continuations were removed from the choice."}]},{"t":"h2","id":"the-tools-all-free","runs":[{"text":"The tools, all free"}]},{"t":"code","lang":"python","label":"Python","text":"from outlines import models, generate\nfrom pydantic import BaseModel\n\nclass Triage(BaseModel):\n    queue: Literal[\"billing\",\"shipping\",\"refunds\",\"technical\",\"other\"]\n    urgency: int\n    summary: str\n\nresult = generate.json(models.transformers(\"Qwen/Qwen2.5-7B-Instruct\"), Triage)(ticket)"},{"t":"p","runs":[{"text":"That returns a validated object, always. Alternatives: "},{"text":"llama.cpp","code":true},{"text":" supports GBNF grammars natively; vLLM has guided decoding built in; XGrammar and lm-format-enforcer provide the same capability with different performance characteristics. Nothing here costs money."}]},{"t":"p","runs":[{"text":"Grammars are more general than JSON schemas. Anything you can express as a context-free grammar (a SQL subset, a domain-specific language, a fixed report template, a limited set of tool calls) can be enforced the same way."}]},{"t":"h2","id":"what-it-guarantees-and-what-it-does-not","runs":[{"text":"What it guarantees, and what it does not"}]},{"t":"p","runs":[{"text":"It guarantees syntax.","b":true},{"text":" The output parses, the enum value is one of your five, the integer is an integer, the required fields are present."}]},{"t":"p","runs":[{"text":"It guarantees nothing about truth.","b":true},{"text":" A constrained decoder will happily emit "},{"text":"{\"queue\": \"billing\", \"urgency\": 3}","code":true},{"text":" for a shipping complaint. Validity and correctness are different properties, and confusing them is the commonest mistake here."}]},{"t":"p","runs":[{"text":"There is also a real debate about whether heavy constraints reduce answer quality. The argument for: forcing a model into a rigid shape from the first token prevents it producing the intermediate reasoning it would otherwise use. The argument against: careful comparisons find little or no degradation when the schema is reasonable and the model is competent. Both positions have supporting experiments, the effect appears to depend heavily on the task and the schema, and it is not settled."}]},{"t":"p","runs":[{"text":"The pragmatic response is a design that sidesteps it: let the model reason freely in a "},{"text":"reasoning","code":true},{"text":" string field that comes "},{"text":"first","i":true},{"text":" in the schema, then constrain the structured fields that follow. You get the thinking and the guarantee."}]},{"t":"h2","id":"the-poor-version-for-hosted-apis","runs":[{"text":"The poor version, for hosted APIs"}]},{"t":"p","runs":[{"text":"If you are calling an API that offers no grammar support, validate and retry:"}]},{"t":"code","lang":"python","label":"Python","text":"for attempt in range(3):\n    out = call_model(prompt)\n    try:\n        return Triage.model_validate_json(out)\n    except ValidationError as e:\n        prompt += f\"\\nThat was invalid: {e}. Return only valid JSON.\"\nraise"},{"t":"p","runs":[{"text":"This works, and it costs a full extra generation every time it fires. At a 4% invalid rate on a million requests that is 40,000 extra calls. Constrained decoding costs nothing per request; retries cost a generation per failure. That arithmetic is how you decide."}]},{"t":"p","runs":[{"text":"Many hosted providers now offer a structured-output mode that does the constraining server-side. Use it where it exists."}]},{"t":"h2","id":"when-fine-tuning-for-format-is-still-right","runs":[{"text":"When fine-tuning for format is still right"}]},{"t":"p","runs":[{"text":"Three cases survive:"}]},{"t":"ol","start":1,"items":[[{"text":"The schema is enormous.","b":true},{"text":" A grammar with hundreds of fields makes the state machine large and the per-token masking expensive. A model that has learned the shape needs less help from the constraint."}],[{"text":"The model must choose the schema.","b":true},{"text":" Constraining requires knowing which grammar to apply. If deciding "},{"text":"which","i":true},{"text":" tool to call, or which of twelve document types this is, is itself the task, that decision cannot be constrained, though it can often be made by a cheap classifier, as module one argued."}],[{"text":"The content inside the structure is wrong.","b":true},{"text":" If the JSON is valid and the values are poor, no constraint helps. That is a knowledge or capability problem, diagnosed as such."}]]},{"t":"h2","id":"the-order-to-try-things","runs":[{"text":"The order to try things"}]},{"t":"ol","start":1,"items":[[{"text":"Constrained decoding. Free, immediate, guaranteed."}],[{"text":"Constrained decoding plus a few examples in the prompt, which improves the "},{"text":"content","i":true},{"text":" inside the valid structure."}],[{"text":"Fine-tuning, if the content is still wrong after both, and then keep the constraint as well, because a fine-tuned model still produces invalid output occasionally and the constraint costs nothing to keep."}]]},{"t":"p","runs":[{"text":"That last point is the one to remember. These are not alternatives. Constrained decoding is a floor you should never remove, and fine-tuning raises what happens above it."}]}],"check":{"question":[{"text":"A team adds JSON-schema constrained decoding and their valid-output rate goes from 91% to 100%, but routing accuracy is unchanged. What does this show?"}],"options":[{"text":[{"text":"The constraint is too permissive; tightening the schema with stricter field types would improve accuracy too."}],"correct":false,"tempting":[{"text":"Assumes accuracy is reachable through the schema, when a stricter schema still cannot distinguish a correct queue from a plausible one."}]},{"text":[{"text":"The constraint fixed syntax, which is not what the accuracy metric measured."}],"correct":true,"tempting":null},{"text":[{"text":"The 9% of outputs that were invalid were also being counted as routing errors, so accuracy should have risen by 9 points."}],"correct":false,"tempting":[{"text":"Assumes the invalid outputs were scored as wrong answers, which would make the unchanged accuracy a contradiction rather than an observation."}]},{"text":[{"text":"Constrained decoding was masking the model’s reasoning tokens, which cancelled out the gain from valid formatting."}],"correct":false,"tempting":[{"text":"Invokes the contested quality-degradation argument to explain a flat result, when the accuracy was simply unaffected in either direction."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"diffusion-lora-and-likeness","title":"LoRA for image models, and the likeness question","minutes":10},"next":{"slug":"small-models-and-edge","title":"Small models, where fine-tuning matters most","minutes":9}}}