{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"structured-output","title":"Structured output, and what a schema cannot promise","url":"https://addaly.com/learn/building-with-ai/structured-output","position":21,"of":80,"minutes":7,"answer":"Schemas guarantee shape, never truth: a perfectly valid object can be perfectly wrong.","keyIdea":"Schemas guarantee shape, never truth: a perfectly valid object can be perfectly wrong."},"body":{"format":"markdown","text":"## Stop parsing prose\n\nThe first version of every extraction feature asks for text and regexes it. It works on Monday. On Thursday the model writes \"Sure, here's the JSON:\" before the JSON, and on Friday it wraps it in a code fence, and by the next week you have 200 lines of cleanup code that is a bad parser.\n\nAsk for a schema instead. Every major provider supports either a JSON schema response format or tool-shaped output, where you declare a function signature and the model fills in the arguments. Both use constrained decoding: the model is prevented, token by token, from emitting anything the schema disallows.\n\n```python\nticket_schema = {\n    \"type\": \"object\",\n    \"properties\": {\n        \"category\": {\"type\": \"string\",\n                     \"enum\": [\"refund\", \"schedule\", \"complaint\", \"other\"]},\n        \"amount_minor_units\": {\"type\": [\"integer\", \"null\"],\n                               \"description\": \"Amount in the smallest unit: paise, kobo, cents. Null if none stated.\"},\n        \"currency\": {\"type\": [\"string\", \"null\"], \"enum\": [\"NGN\", \"INR\", \"USD\", \"EUR\", None]},\n        \"urgent\": {\"type\": \"boolean\"},\n    },\n    \"required\": [\"category\", \"urgent\"],\n    \"additionalProperties\": False,\n}\n```\n\nTwo choices in there are worth copying. Money is an integer in minor units, never a float: 4500 kobo, not 45.0. And every enum has an escape value, because a required enum with no way out forces the model to pick something.\n\n## Validate anyway\n\nConstrained decoding is good, not perfect, and your schema will drift from your code. Parse into a typed object and let the failure be loud.\n\n```python\nimport json\nfrom pydantic import BaseModel, ValidationError\n\nclass Ticket(BaseModel):\n    category: str\n    amount_minor_units: int | None = None\n    currency: str | None = None\n    urgent: bool\n\ndef extract(email: str, tries: int = 2) -> Ticket:\n    messages = [{\"role\": \"user\", \"content\": email}]\n    for _ in range(tries):\n        raw = call_model_with_schema(messages, ticket_schema)\n        try:\n            return Ticket.model_validate(raw)\n        except ValidationError as e:\n            messages += [\n                {\"role\": \"assistant\", \"content\": json.dumps(raw)},\n                {\"role\": \"user\", \"content\": f\"That failed validation:\\n{e}\\nReturn corrected JSON only.\"},\n            ]\n    raise RuntimeError(\"no valid extraction after retries\")\n```\n\nHanding the validation error back is the cheapest repair loop there is, and it usually works on the first retry. Cap the retries. An unbounded repair loop is a bill.\n\n## The honest part: shape is not truth\n\nHere is the thing teams miss, often for months. A schema guarantees that a field exists and has the right type. It says nothing about whether the value is correct.\n\n`{\"amount_minor_units\": 4500}` is a perfectly valid object when the invoice said 450. Parse errors go to zero, so the feature looks fixed, and the wrong values now sail straight into your database with no exception to catch them.\n\nWorse, `required` actively encourages this. A required field must be emitted. If the document does not contain an invoice number, the model has to write something, and something is what you get. This is one of the most reliable ways to manufacture hallucinations by accident.\n\nSo:\n\n- Mark fields optional and nullable unless they truly must be present.\n- Give every enum an `other` or `unknown` member.\n- Add a `source_quote` field holding the exact text the value came from. It costs a few tokens and turns a silent wrong number into something you can check, by eye or in code.\n\n```python\n\"source_quote\": {\"type\": [\"string\", \"null\"],\n                 \"description\": \"Exact substring of the input supporting the amount. Null if not stated.\"}\n```\n\nThen assert in code that `source_quote` is a substring of the input. That single check catches a surprising share of invented values, and it is deterministic: no second model call required.\n\n## Do not ask for confidence\n\nA tempting field is `\"confidence\": {\"type\": \"number\"}`. Resist it. Self-reported confidence from a language model is poorly calibrated: it clusters around 0.9, and it is high on exactly the fluent, plausible errors you most want to catch. If you need a confidence signal, derive it from something real: did the quote match, do two runs agree, did retrieval find a supporting document.\n\n> Figure: One thousand invoices through a constrained schema. A two by two grid, rows Passed the schema and Failed the schema, columns Amount correct and Amount wrong: Passed the schema and Amount correct: 912; Passed the schema and Amount wrong: 76; Failed the schema and Amount correct: 0; Failed the schema and Amount wrong: 12. Constrained decoding took parse failures to almost nothing, so the feature looks fixed. The 76 wrong amounts did not go away; they lost the exception that used to catch them and now reach the database silently. A required source_quote, checked as a substring in code, is what puts a check back in the top-right cell."},"check":{"question":"A team switches an invoice extractor to strict schema output. Parse errors drop from about 40 a day to zero. Two weeks later, finance finds a batch of invoices with the wrong totals recorded. What did the schema actually buy them?","options":[{"text":"The schema is too loose; tightening the types and adding format constraints would fix the totals as well.","correct":false,"tempting":"Types did fix the crashes, so more types feels like it must mean more correctness."},{"text":"Constrained decoding reduces hallucination, so the remaining errors must come from bad input such as OCR noise.","correct":false,"tempting":"Constrained decoding really does eliminate format drift, and it is a small step to assume it constrains facts too."},{"text":"It guarantees the shape of the output, not the content: every field is present and correctly typed, and can still hold the wrong value.","correct":true},{"text":"Required fields force the model to locate the value, so a wrong total means the number was genuinely absent from the document.","correct":false,"tempting":"'Required' sounds like a promise that the data was found, rather than a promise that a field was emitted no matter what."}]},"prev":{"slug":"when-prompting-stops-working","title":"When prompting stops working: the fine-tuning decision","url":"https://addaly.com/learn/building-with-ai/when-prompting-stops-working"},"next":{"slug":"schemas-models-can-follow","title":"Designing a schema a model can actually fill","url":"https://addaly.com/learn/building-with-ai/schemas-models-can-follow"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Structured output, and what a schema cannot promise', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/structured-output (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Structured output, and what a schema cannot promise","courseTitle":"Building with AI","module":{"index":3,"slug":"structure-tools-and-the-loop","title":"Structure, tools and the loop"},"answer":null,"keyIdea":"Schemas guarantee shape, never truth: a perfectly valid object can be perfectly wrong.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 3.1","figure":{"cols":["Amount correct","Amount wrong"],"rows":["Passed the schema","Failed the schema"],"type":"matrix","cells":[["912","76"],["0","12"]],"title":"One thousand invoices through a constrained schema","caption":"Constrained decoding took parse failures to almost nothing, so the feature looks fixed. The 76 wrong amounts did not go away; they lost the exception that used to catch them and now reach the database silently. A required source_quote, checked as a substring in code, is what puts a check back in the top-right cell.","cellLabels":[["auto-accepted, rightly","auto-accepted, wrongly"],["cannot happen","caught by the validator"]]},"alt":"Figure: One thousand invoices through a constrained schema. A two by two grid, rows Passed the schema and Failed the schema, columns Amount correct and Amount wrong: Passed the schema and Amount correct: 912; Passed the schema and Amount wrong: 76; Failed the schema and Amount correct: 0; Failed the schema and Amount wrong: 12. Constrained decoding took parse failures to almost nothing, so the feature looks fixed. The 76 wrong amounts did not go away; they lost the exception that used to catch them and now reach the database silently. A required source_quote, checked as a substring in code, is what puts a check back in the top-right cell."}},"blocks":[{"t":"h2","id":"stop-parsing-prose","runs":[{"text":"Stop parsing prose"}]},{"t":"p","runs":[{"text":"The first version of every extraction feature asks for text and regexes it. It works on Monday. On Thursday the model writes “Sure, here’s the JSON:” before the JSON, and on Friday it wraps it in a code fence, and by the next week you have 200 lines of cleanup code that is a bad parser."}]},{"t":"p","runs":[{"text":"Ask for a schema instead. Every major provider supports either a JSON schema response format or tool-shaped output, where you declare a function signature and the model fills in the arguments. Both use constrained decoding: the model is prevented, token by token, from emitting anything the schema disallows."}]},{"t":"code","lang":"python","label":"Python","text":"ticket_schema = {\n    \"type\": \"object\",\n    \"properties\": {\n        \"category\": {\"type\": \"string\",\n                     \"enum\": [\"refund\", \"schedule\", \"complaint\", \"other\"]},\n        \"amount_minor_units\": {\"type\": [\"integer\", \"null\"],\n                               \"description\": \"Amount in the smallest unit: paise, kobo, cents. Null if none stated.\"},\n        \"currency\": {\"type\": [\"string\", \"null\"], \"enum\": [\"NGN\", \"INR\", \"USD\", \"EUR\", None]},\n        \"urgent\": {\"type\": \"boolean\"},\n    },\n    \"required\": [\"category\", \"urgent\"],\n    \"additionalProperties\": False,\n}"},{"t":"p","runs":[{"text":"Two choices in there are worth copying. Money is an integer in minor units, never a float: 4500 kobo, not 45.0. And every enum has an escape value, because a required enum with no way out forces the model to pick something."}]},{"t":"h2","id":"validate-anyway","runs":[{"text":"Validate anyway"}]},{"t":"p","runs":[{"text":"Constrained decoding is good, not perfect, and your schema will drift from your code. Parse into a typed object and let the failure be loud."}]},{"t":"code","lang":"python","label":"Python","text":"import json\nfrom pydantic import BaseModel, ValidationError\n\nclass Ticket(BaseModel):\n    category: str\n    amount_minor_units: int | None = None\n    currency: str | None = None\n    urgent: bool\n\ndef extract(email: str, tries: int = 2) -> Ticket:\n    messages = [{\"role\": \"user\", \"content\": email}]\n    for _ in range(tries):\n        raw = call_model_with_schema(messages, ticket_schema)\n        try:\n            return Ticket.model_validate(raw)\n        except ValidationError as e:\n            messages += [\n                {\"role\": \"assistant\", \"content\": json.dumps(raw)},\n                {\"role\": \"user\", \"content\": f\"That failed validation:\\n{e}\\nReturn corrected JSON only.\"},\n            ]\n    raise RuntimeError(\"no valid extraction after retries\")"},{"t":"p","runs":[{"text":"Handing the validation error back is the cheapest repair loop there is, and it usually works on the first retry. Cap the retries. An unbounded repair loop is a bill."}]},{"t":"h2","id":"the-honest-part-shape-is-not-truth","runs":[{"text":"The honest part: shape is not truth"}]},{"t":"p","runs":[{"text":"Here is the thing teams miss, often for months. A schema guarantees that a field exists and has the right type. It says nothing about whether the value is correct."}]},{"t":"p","runs":[{"text":"{\"amount_minor_units\": 4500}","code":true},{"text":" is a perfectly valid object when the invoice said 450. Parse errors go to zero, so the feature looks fixed, and the wrong values now sail straight into your database with no exception to catch them."}]},{"t":"p","runs":[{"text":"Worse, "},{"text":"required","code":true},{"text":" actively encourages this. A required field must be emitted. If the document does not contain an invoice number, the model has to write something, and something is what you get. This is one of the most reliable ways to manufacture hallucinations by accident."}]},{"t":"p","runs":[{"text":"So:"}]},{"t":"ul","items":[[{"text":"Mark fields optional and nullable unless they truly must be present."}],[{"text":"Give every enum an "},{"text":"other","code":true},{"text":" or "},{"text":"unknown","code":true},{"text":" member."}],[{"text":"Add a "},{"text":"source_quote","code":true},{"text":" field holding the exact text the value came from. It costs a few tokens and turns a silent wrong number into something you can check, by eye or in code."}]]},{"t":"code","lang":"python","label":"Python","text":"\"source_quote\": {\"type\": [\"string\", \"null\"],\n                 \"description\": \"Exact substring of the input supporting the amount. Null if not stated.\"}"},{"t":"p","runs":[{"text":"Then assert in code that "},{"text":"source_quote","code":true},{"text":" is a substring of the input. That single check catches a surprising share of invented values, and it is deterministic: no second model call required."}]},{"t":"h2","id":"do-not-ask-for-confidence","runs":[{"text":"Do not ask for confidence"}]},{"t":"p","runs":[{"text":"A tempting field is "},{"text":"\"confidence\": {\"type\": \"number\"}","code":true},{"text":". Resist it. Self-reported confidence from a language model is poorly calibrated: it clusters around 0.9, and it is high on exactly the fluent, plausible errors you most want to catch. If you need a confidence signal, derive it from something real: did the quote match, do two runs agree, did retrieval find a supporting document."}]}],"check":{"question":[{"text":"A team switches an invoice extractor to strict schema output. Parse errors drop from about 40 a day to zero. Two weeks later, finance finds a batch of invoices with the wrong totals recorded. What did the schema actually buy them?"}],"options":[{"text":[{"text":"The schema is too loose; tightening the types and adding format constraints would fix the totals as well."}],"correct":false,"tempting":[{"text":"Types did fix the crashes, so more types feels like it must mean more correctness."}]},{"text":[{"text":"Constrained decoding reduces hallucination, so the remaining errors must come from bad input such as OCR noise."}],"correct":false,"tempting":[{"text":"Constrained decoding really does eliminate format drift, and it is a small step to assume it constrains facts too."}]},{"text":[{"text":"It guarantees the shape of the output, not the content: every field is present and correctly typed, and can still hold the wrong value."}],"correct":true,"tempting":null},{"text":[{"text":"Required fields force the model to locate the value, so a wrong total means the number was genuinely absent from the document."}],"correct":false,"tempting":[{"text":"‘Required’ sounds like a promise that the data was found, rather than a promise that a field was emitted no matter what."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"when-prompting-stops-working","title":"When prompting stops working: the fine-tuning decision","minutes":10},"next":{"slug":"schemas-models-can-follow","title":"Designing a schema a model can actually fill","minutes":9}}}