Validating what the model returns: from a string that looks like JSON to an object you can trust

It is a string

Whatever a model API returns as "the answer" is text. Even when you asked for JSON, even when the provider offers a JSON mode, what arrives in content is a str, and until you have parsed it and checked its shape you do not have data; you have a string that resembles data. Every failure in this lesson comes from treating the resemblance as the thing.

The first parse

import json

raw = reply_text
try:
    data = json.loads(raw)
except json.JSONDecodeError as e:
    raise ValueError(f"model did not return JSON: {e}; got {raw[:200]!r}")

Three things break this in practice.

Fences. Models like to wrap JSON in a Markdown code block: three backticks, the word json, the object, three backticks. json.loads sees the backticks and fails at character 0. Strip them before parsing:

def strip_fences(s):
    s = s.strip()
    if s.startswith("```"):
        s = s.split("\n", 1)[1] if "\n" in s else s[3:]
        if s.endswith("```"):
            s = s[:-3]
    return s.strip()

Preamble. "Here is the JSON you asked for:" followed by the object. Find the first { and the last } and parse what lies between. That is a heuristic; it fails on text containing braces. Provider JSON modes exist to remove the need for it, and when one is available you should use it and still keep the parse inside a try.

Truncation. A max_tokens too small for the answer cuts the JSON mid-string, and the parse fails with "Unterminated string". The fix is not in the parser; check the stop reason in the response before parsing, and raise a distinct error when it says length.

Parsing is not validating

json.loads succeeding means the text was valid JSON. It says nothing about whether the JSON has the fields you need, with the types you need. This parses:

{"name": "Widget", "price": "12.50", "in_stock": "yes"}

Your code expects price to be a number and in_stock a boolean. The string "12.50" will flow through three functions and fail in a comparison far from where it entered. The string "yes" is truthy, so if item["in_stock"]: is always true, and a product marked "no" is also in stock. That one never crashes at all.

pydantic: the check in one line

from pydantic import BaseModel, ValidationError

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool

try:
    product = Product.model_validate_json(strip_fences(raw))
except ValidationError as e:
    print(e)

model_validate_json parses and validates in one step. The model reply above produces:

1 validation error for Product
in_stock
  Input should be a valid boolean, unable to interpret input [type=bool_parsing, input_value='yes', input_type=str]

Note what happened to price: "12.50" was coerced to 12.5, because pydantic in its default mode converts a numeric string to a float. "yes" was not coerced to a boolean because pydantic only accepts a fixed set of strings for booleans ("true", "false", "1", "0", "yes", "no" and a few more — and "yes" is in that set in recent versions, so check which version you have and test the case). If you want no coercion at all, model_config = ConfigDict(strict=True) rejects anything that is not already the right type.

Constrain further with types that carry rules:

from typing import Literal
from pydantic import Field

class Product(BaseModel):
    name: str = Field(min_length=1)
    price: float = Field(ge=0)
    in_stock: bool
    category: Literal["tool", "part", "consumable"]

Now a negative price, an empty name or a category the model invented all fail at the boundary with a message naming the field. A list of products is list[Product], or a wrapper model with a products: list[Product] field, which is usually safer because it survives the model returning an object where you expected an array.

Feed the error back

The validation error is written for humans, and models read it well. The cheapest repair loop:

for attempt in range(3):
    raw = ask(messages)
    try:
        return Product.model_validate_json(strip_fences(raw))
    except ValidationError as e:
        messages.append({"role": "assistant", "content": raw})
        messages.append({"role": "user",
                         "content": f"That did not validate:\n{e}\nReturn only corrected JSON."})
raise RuntimeError("no valid output after 3 attempts")

The second attempt succeeds most of the time. Cap the attempts: each one costs money, and an output that fails three times is telling you the prompt or the schema is wrong.

The schema in the prompt

pydantic can print the JSON Schema for a model, and the provider's structured-output features accept that schema directly:

print(json.dumps(Product.model_json_schema(), indent=2))

Putting that in the prompt, or in the response_format parameter where supported, tells the model the exact shape. The validation on your side stays. A schema in the prompt makes valid output likely; the check makes invalid output impossible to act on.

What validation cannot do

It confirms shape, not truth. A price of 12.5 for a product that costs 125 validates perfectly. A category chosen from the allowed three can still be the wrong one. Validation is the floor: it guarantees the rest of your program receives the types it was written for. Whether the values are right is a question for evaluation, which is its own course.

Try this now

Ask any model — a local one is fine — for a JSON list of three products with those four fields, and run it through list[Product] validation ten times. Count how many replies needed the fence stripper, how many failed validation, and on which field. Those counts are your prompt's real reliability, and they are never zero on the first try.