{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"validating-what-the-model-returns","title":"Validating what the model returns: from a string that looks like JSON to an object you can trust","url":"https://addaly.com/learn/python-for-ai/validating-what-the-model-returns","position":58,"of":89,"minutes":9,"answer":"A model's reply is text until you have parsed it and checked every field against a schema; pydantic turns that check into one line.","keyIdea":"A model's reply is text until you have parsed it and checked every field against a schema; pydantic turns that check into one line, and feeding its error message back to the model is the cheapest repair there is."},"body":{"format":"markdown","text":"## It is a string\n\nWhatever a model API returns as \"the answer\" is text. Even when you asked for JSON, even when the provider offers a JSON mode, what arrives in `content` is a `str`, and until you have parsed it and checked its shape you do not have data; you have a string that resembles data. Every failure in this lesson comes from treating the resemblance as the thing.\n\n## The first parse\n\n```python\nimport json\n\nraw = reply_text\ntry:\n    data = json.loads(raw)\nexcept json.JSONDecodeError as e:\n    raise ValueError(\n        (\n            f\"model did not return JSON: {e}; \"\n            f\"got {raw[:200]!r}\"\n        )\n    )\n```\n\nThree things break this in practice.\n\n**Fences.** Models like to wrap JSON in a Markdown code block: three backticks, the word `json`, the object, three backticks. `json.loads` sees the backticks and fails at character 0. Strip them before parsing:\n\n```python\ndef strip_fences(s):\n    s = s.strip()\n    if s.startswith(\"```\"):\n        s = s.split(\"\\n\", 1)[1] if \"\\n\" in s else s[3:]\n        if s.endswith(\"```\"):\n            s = s[:-3]\n    return s.strip()\n```\n\n**Preamble.** \"Here is the JSON you asked for:\" followed by the object. Find the first `{` and the last `}` and parse what lies between. That is a heuristic; it fails on text containing braces. Provider JSON modes exist to remove the need for it, and when one is available you should use it and still keep the parse inside a `try`.\n\n**Truncation.** A `max_tokens` too small for the answer cuts the JSON mid-string, and the parse fails with \"Unterminated string\". The fix is not in the parser; check the stop reason in the response before parsing, and raise a distinct error when it says `length`.\n\n## Parsing is not validating\n\n`json.loads` succeeding means the text was valid JSON. It says nothing about whether the JSON has the fields you need, with the types you need. This parses:\n\n```python\n{\"name\": \"Widget\", \"price\": \"12.50\", \"in_stock\": \"yes\"}\n```\n\nYour code expects `price` to be a number and `in_stock` a boolean. The string `\"12.50\"` will flow through three functions and fail in a comparison far from where it entered. The string `\"yes\"` is truthy, so `if item[\"in_stock\"]:` is always true, and a product marked `\"no\"` is also in stock. That one never crashes at all.\n\n## pydantic: the check in one line\n\n```python\nfrom pydantic import BaseModel, ValidationError\n\nclass Product(BaseModel):\n    name: str\n    price: float\n    in_stock: bool\n\ntry:\n    product = Product.model_validate_json(strip_fences(raw))\nexcept ValidationError as e:\n    print(e)\n```\n\n`model_validate_json` parses and validates in one step. The model reply above passes and prints nothing: `product` is `Product(name='Widget', price=12.5, in_stock=True)`. Change `\"yes\"` to `\"in stock\"` and it produces:\n\n```\n1 validation error for Product\nin_stock\n  Input should be a valid boolean, unable to interpret input [type=bool_parsing, input_value='in stock', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.14/v/bool_parsing\n```\n\nNote what happened to `price`: `\"12.50\"` was **coerced** to `12.5`, because pydantic in its default mode converts a numeric string to a float. `\"yes\"` was coerced too, to `True`, and `\"no\"` would become `False`, which fixes the always-in-stock bug from the last section: pydantic accepts a fixed set of strings for booleans (`\"true\"`, `\"false\"`, `\"1\"`, `\"0\"`, `\"yes\"`, `\"no\"`, `\"on\"`, `\"off\"` and a few more, in any case, and has since version 1.0). `\"in stock\"` is not in that set, so it is rejected. If you want no coercion at all, `model_config = ConfigDict(strict=True)` rejects anything that is not already the right type.\n\nConstrain further with types that carry rules:\n\n```python\nfrom typing import Literal\nfrom pydantic import Field\n\nclass Product(BaseModel):\n    name: str = Field(min_length=1)\n    price: float = Field(ge=0)\n    in_stock: bool\n    category: Literal[\"tool\", \"part\", \"consumable\"]\n```\n\nNow a negative price, an empty name or a category the model invented all fail at the boundary with a message naming the field. A list of products is `list[Product]`, or a wrapper model with a `products: list[Product]` field, which is usually safer because it survives the model returning an object where you expected an array.\n\n## Feed the error back\n\nThe validation error is written for humans, and models read it well. The cheapest repair loop:\n\n```python\nfor attempt in range(3):\n    raw = ask(messages)\n    try:\n        return Product.model_validate_json(\n            strip_fences(raw)\n        )\n    except ValidationError as e:\n        messages.append(\n            {\"role\": \"assistant\", \"content\": raw}\n        )\n        messages.append(\n            {\n                \"role\": \"user\",\n                \"content\": (\n                    f\"That did not validate:\\n\"\n                    f\"{e}\"\n                    f\"\\nReturn only corrected JSON.\"\n                ),\n            }\n        )\nraise RuntimeError(\"no valid output after 3 attempts\")\n```\n\nThe second attempt succeeds most of the time. Cap the attempts: each one costs money, and an output that fails three times is telling you the prompt or the schema is wrong.\n\n## The schema in the prompt\n\npydantic can print the JSON Schema for a model, and the provider's structured-output features accept that schema directly:\n\n```python\nprint(json.dumps(Product.model_json_schema(), indent=2))\n```\n\nPutting that in the prompt, or in the `response_format` parameter where supported, tells the model the exact shape. The validation on your side stays. A schema in the prompt makes valid output likely; the check makes invalid output impossible to act on.\n\n## What validation cannot do\n\nIt confirms shape, not truth. A price of `12.5` for a product that costs `125` validates perfectly. A category chosen from the allowed three can still be the wrong one. Validation is the floor: it guarantees the rest of your program receives the types it was written for. Whether the values are right is a question for evaluation, which is its own course.\n\n## Try this now\n\nAsk any model (a local one is fine) for a JSON list of three products with those four fields, and run it through `list[Product]` validation ten times. Count how many replies needed the fence stripper, how many failed validation, and on which field. Those counts are your prompt's real reliability, and they are never zero on the first try."},"check":{"question":"A function asks a model for JSON with fields `name`, `price` and `in_stock`, parses the reply with `json.loads`, and returns the dict. Downstream code later crashes with `TypeError: '<' not supported between instances of 'str' and 'int'` while comparing prices. Which change addresses the actual cause?","options":[{"text":"Validate the parsed reply at the boundary against a schema declaring `price: float`, so a string price is coerced or rejected there, not three functions later.","correct":true},{"text":"Wrap the comparison in `try/except TypeError` and skip items that fail, since some products have no price and the model cannot be expected to invent one for them.","correct":false,"tempting":"This hides the symptom; the price is present but arrived as a string, and skipping it loses real data for a fixable reason."},{"text":"Ask the model to return the price as an integer by adding \"return numbers as numbers\" to the prompt.","correct":false,"tempting":"Prompting reduces how often it happens but cannot guarantee the type; the code still has to handle the case it does not."},{"text":"Replace `json.loads` with `ast.literal_eval`, which converts numeric strings to numbers automatically.","correct":false,"tempting":"`literal_eval` parses Python literals, not JSON, and does not convert a quoted string into a number either."}]},"prev":{"slug":"threads-processes-and-the-gil","title":"Threads, processes, and the lock that decides which one you need","url":"https://addaly.com/learn/python-for-ai/threads-processes-and-the-gil"},"next":{"slug":"counting-tokens-and-cost","title":"Knowing what each call cost: usage figures, token counting, and a budget that stops the program","url":"https://addaly.com/learn/python-for-ai/counting-tokens-and-cost"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Validating what the model returns: from a string that looks like JSON to an object you can trust', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/validating-what-the-model-returns (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Validating what the model returns: from a string that looks like JSON to an object you can trust","courseTitle":"Python, from Zero, for AI","module":{"index":6,"slug":"talking-to-a-service","title":"Talking to a service: HTTP from Python, done properly"},"answer":null,"keyIdea":"A model’s reply is text until you have parsed it and checked every field against a schema; pydantic turns that check into one line, and feeding its error message back to the model is the cheapest repair there is.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"it-is-a-string","runs":[{"text":"It is a string"}]},{"t":"p","runs":[{"text":"Whatever a model API returns as “the answer” is text. Even when you asked for JSON, even when the provider offers a JSON mode, what arrives in "},{"text":"content","code":true},{"text":" is a "},{"text":"str","code":true},{"text":", and until you have parsed it and checked its shape you do not have data; you have a string that resembles data. Every failure in this lesson comes from treating the resemblance as the thing."}]},{"t":"h2","id":"the-first-parse","runs":[{"text":"The first parse"}]},{"t":"code","lang":"python","label":"Python","text":"import json\n\nraw = reply_text\ntry:\n    data = json.loads(raw)\nexcept json.JSONDecodeError as e:\n    raise ValueError(\n        (\n            f\"model did not return JSON: {e}; \"\n            f\"got {raw[:200]!r}\"\n        )\n    )"},{"t":"p","runs":[{"text":"Three things break this in practice."}]},{"t":"p","runs":[{"text":"Fences.","b":true},{"text":" Models like to wrap JSON in a Markdown code block: three backticks, the word "},{"text":"json","code":true},{"text":", the object, three backticks. "},{"text":"json.loads","code":true},{"text":" sees the backticks and fails at character 0. Strip them before parsing:"}]},{"t":"code","lang":"python","label":"Python","text":"def strip_fences(s):\n    s = s.strip()\n    if s.startswith(\"```\"):\n        s = s.split(\"\\n\", 1)[1] if \"\\n\" in s else s[3:]\n        if s.endswith(\"```\"):\n            s = s[:-3]\n    return s.strip()"},{"t":"p","runs":[{"text":"Preamble.","b":true},{"text":" “Here is the JSON you asked for:” followed by the object. Find the first "},{"text":"{","code":true},{"text":" and the last "},{"text":"}","code":true},{"text":" and parse what lies between. That is a heuristic; it fails on text containing braces. Provider JSON modes exist to remove the need for it, and when one is available you should use it and still keep the parse inside a "},{"text":"try","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Truncation.","b":true},{"text":" A "},{"text":"max_tokens","code":true},{"text":" too small for the answer cuts the JSON mid-string, and the parse fails with “Unterminated string”. The fix is not in the parser; check the stop reason in the response before parsing, and raise a distinct error when it says "},{"text":"length","code":true},{"text":"."}]},{"t":"h2","id":"parsing-is-not-validating","runs":[{"text":"Parsing is not validating"}]},{"t":"p","runs":[{"text":"json.loads","code":true},{"text":" succeeding means the text was valid JSON. It says nothing about whether the JSON has the fields you need, with the types you need. This parses:"}]},{"t":"code","lang":"python","label":"Python","text":"{\"name\": \"Widget\", \"price\": \"12.50\", \"in_stock\": \"yes\"}"},{"t":"p","runs":[{"text":"Your code expects "},{"text":"price","code":true},{"text":" to be a number and "},{"text":"in_stock","code":true},{"text":" a boolean. The string "},{"text":"\"12.50\"","code":true},{"text":" will flow through three functions and fail in a comparison far from where it entered. The string "},{"text":"\"yes\"","code":true},{"text":" is truthy, so "},{"text":"if item[\"in_stock\"]:","code":true},{"text":" is always true, and a product marked "},{"text":"\"no\"","code":true},{"text":" is also in stock. That one never crashes at all."}]},{"t":"h2","id":"pydantic-the-check-in-one-line","runs":[{"text":"pydantic: the check in one line"}]},{"t":"code","lang":"python","label":"Python","text":"from pydantic import BaseModel, ValidationError\n\nclass Product(BaseModel):\n    name: str\n    price: float\n    in_stock: bool\n\ntry:\n    product = Product.model_validate_json(strip_fences(raw))\nexcept ValidationError as e:\n    print(e)"},{"t":"p","runs":[{"text":"model_validate_json","code":true},{"text":" parses and validates in one step. The model reply above passes and prints nothing: "},{"text":"product","code":true},{"text":" is "},{"text":"Product(name='Widget', price=12.5, in_stock=True)","code":true},{"text":". Change "},{"text":"\"yes\"","code":true},{"text":" to "},{"text":"\"in stock\"","code":true},{"text":" and it produces:"}]},{"t":"code","lang":"","label":"Text","text":"1 validation error for Product\nin_stock\n  Input should be a valid boolean, unable to interpret input [type=bool_parsing, input_value='in stock', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.14/v/bool_parsing"},{"t":"p","runs":[{"text":"Note what happened to "},{"text":"price","code":true},{"text":": "},{"text":"\"12.50\"","code":true},{"text":" was "},{"text":"coerced","b":true},{"text":" to "},{"text":"12.5","code":true},{"text":", because pydantic in its default mode converts a numeric string to a float. "},{"text":"\"yes\"","code":true},{"text":" was coerced too, to "},{"text":"True","code":true},{"text":", and "},{"text":"\"no\"","code":true},{"text":" would become "},{"text":"False","code":true},{"text":", which fixes the always-in-stock bug from the last section: pydantic accepts a fixed set of strings for booleans ("},{"text":"\"true\"","code":true},{"text":", "},{"text":"\"false\"","code":true},{"text":", "},{"text":"\"1\"","code":true},{"text":", "},{"text":"\"0\"","code":true},{"text":", "},{"text":"\"yes\"","code":true},{"text":", "},{"text":"\"no\"","code":true},{"text":", "},{"text":"\"on\"","code":true},{"text":", "},{"text":"\"off\"","code":true},{"text":" and a few more, in any case, and has since version 1.0). "},{"text":"\"in stock\"","code":true},{"text":" is not in that set, so it is rejected. If you want no coercion at all, "},{"text":"model_config = ConfigDict(strict=True)","code":true},{"text":" rejects anything that is not already the right type."}]},{"t":"p","runs":[{"text":"Constrain further with types that carry rules:"}]},{"t":"code","lang":"python","label":"Python","text":"from typing import Literal\nfrom pydantic import Field\n\nclass Product(BaseModel):\n    name: str = Field(min_length=1)\n    price: float = Field(ge=0)\n    in_stock: bool\n    category: Literal[\"tool\", \"part\", \"consumable\"]"},{"t":"p","runs":[{"text":"Now a negative price, an empty name or a category the model invented all fail at the boundary with a message naming the field. A list of products is "},{"text":"list[Product]","code":true},{"text":", or a wrapper model with a "},{"text":"products: list[Product]","code":true},{"text":" field, which is usually safer because it survives the model returning an object where you expected an array."}]},{"t":"h2","id":"feed-the-error-back","runs":[{"text":"Feed the error back"}]},{"t":"p","runs":[{"text":"The validation error is written for humans, and models read it well. The cheapest repair loop:"}]},{"t":"code","lang":"python","label":"Python","text":"for attempt in range(3):\n    raw = ask(messages)\n    try:\n        return Product.model_validate_json(\n            strip_fences(raw)\n        )\n    except ValidationError as e:\n        messages.append(\n            {\"role\": \"assistant\", \"content\": raw}\n        )\n        messages.append(\n            {\n                \"role\": \"user\",\n                \"content\": (\n                    f\"That did not validate:\\n\"\n                    f\"{e}\"\n                    f\"\\nReturn only corrected JSON.\"\n                ),\n            }\n        )\nraise RuntimeError(\"no valid output after 3 attempts\")"},{"t":"p","runs":[{"text":"The second attempt succeeds most of the time. Cap the attempts: each one costs money, and an output that fails three times is telling you the prompt or the schema is wrong."}]},{"t":"h2","id":"the-schema-in-the-prompt","runs":[{"text":"The schema in the prompt"}]},{"t":"p","runs":[{"text":"pydantic can print the JSON Schema for a model, and the provider’s structured-output features accept that schema directly:"}]},{"t":"code","lang":"python","label":"Python","text":"print(json.dumps(Product.model_json_schema(), indent=2))"},{"t":"p","runs":[{"text":"Putting that in the prompt, or in the "},{"text":"response_format","code":true},{"text":" parameter where supported, tells the model the exact shape. The validation on your side stays. A schema in the prompt makes valid output likely; the check makes invalid output impossible to act on."}]},{"t":"h2","id":"what-validation-cannot-do","runs":[{"text":"What validation cannot do"}]},{"t":"p","runs":[{"text":"It confirms shape, not truth. A price of "},{"text":"12.5","code":true},{"text":" for a product that costs "},{"text":"125","code":true},{"text":" validates perfectly. A category chosen from the allowed three can still be the wrong one. Validation is the floor: it guarantees the rest of your program receives the types it was written for. Whether the values are right is a question for evaluation, which is its own course."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Ask any model (a local one is fine) for a JSON list of three products with those four fields, and run it through "},{"text":"list[Product]","code":true},{"text":" validation ten times. Count how many replies needed the fence stripper, how many failed validation, and on which field. Those counts are your prompt’s real reliability, and they are never zero on the first try."}]}],"check":{"question":[{"text":"A function asks a model for JSON with fields "},{"text":"name","code":true},{"text":", "},{"text":"price","code":true},{"text":" and "},{"text":"in_stock","code":true},{"text":", parses the reply with "},{"text":"json.loads","code":true},{"text":", and returns the dict. Downstream code later crashes with "},{"text":"TypeError: '<' not supported between instances of 'str' and 'int'","code":true},{"text":" while comparing prices. Which change addresses the actual cause?"}],"options":[{"text":[{"text":"Validate the parsed reply at the boundary against a schema declaring "},{"text":"price: float","code":true},{"text":", so a string price is coerced or rejected there, not three functions later."}],"correct":true,"tempting":null},{"text":[{"text":"Wrap the comparison in "},{"text":"try/except TypeError","code":true},{"text":" and skip items that fail, since some products have no price and the model cannot be expected to invent one for them."}],"correct":false,"tempting":[{"text":"This hides the symptom; the price is present but arrived as a string, and skipping it loses real data for a fixable reason."}]},{"text":[{"text":"Ask the model to return the price as an integer by adding “return numbers as numbers” to the prompt."}],"correct":false,"tempting":[{"text":"Prompting reduces how often it happens but cannot guarantee the type; the code still has to handle the case it does not."}]},{"text":[{"text":"Replace "},{"text":"json.loads","code":true},{"text":" with "},{"text":"ast.literal_eval","code":true},{"text":", which converts numeric strings to numbers automatically."}],"correct":false,"tempting":[{"text":"literal_eval","code":true},{"text":" parses Python literals, not JSON, and does not convert a quoted string into a number either."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"threads-processes-and-the-gil","title":"Threads, processes, and the lock that decides which one you need","minutes":9},"next":{"slug":"counting-tokens-and-cost","title":"Knowing what each call cost: usage figures, token counting, and a budget that stops the program","minutes":9}}}