{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"structured-sources-in-context","title":"Rows, records and schemas in the window","url":"https://addaly.com/learn/context-engineering/structured-sources-in-context","position":27,"of":87,"minutes":9,"answer":"Structured data reaches the window as records, as a schema the model writes a query against, or as the result of a function you wrote: choose by whether the answer is a lookup.","keyIdea":"Structured data reaches the window as records, as a schema the model writes a query against, or as the result of a function you wrote: choose by whether the answer is a lookup, an aggregate or a known operation, and never let generated SQL run with more than read permission on named views."},"body":{"format":"markdown","text":"## Not everything worth retrieving is a document\n\nThe question is \"how many of my orders shipped late last month\". No amount of chunking and embedding will answer it, because the answer is a count over rows that no document contains. Somewhere between a third and a half of the questions people ask a business assistant are like this, and a pipeline built entirely around passage retrieval fails all of them in the same confident way.\n\nStructured data enters the window through three different doors, and choosing the right one is most of the work.\n\n## Door one: the row goes in the context\n\nSmall, specific results. The customer's order. Their current plan. Three open tickets.\n\nSerialise them so each record stands alone:\n\n```\norder=ORD-88213 status=shipped placed=2026-08-14 delivered=2026-08-19\n  items=2 total=4,180 INR carrier=Bluedart\n```\n\nRepeating the field names on every record costs tokens and buys robustness: the record survives being split, reordered or quoted in isolation. For a handful of records that trade is clearly right.\n\nAbove roughly twenty records it inverts, and a header-once format is much cheaper: the field names stop being worth their repetition. State the header immediately above the block so nothing can separate them.\n\nTwo rules regardless of format. **Include units and currency on every number**, because `4180` and `4,180 INR` are different facts and the model will guess. And **state the row count and whether the set is complete**: \"12 of 87 orders, most recent first\". Without it the model treats twelve as all of them, which is the same partial-set failure the previous module described.\n\n## Door two: the schema goes in the context, and the model writes a query\n\nFor aggregates and joins (counts, sums, groupings) the data cannot go in the window and should not. Put the schema in and let the model produce SQL.\n\nWhat the schema block needs, in order of value:\n\n1. **Table and column names**, with types.\n2. **A one-line description per column** where the name is not self-explanatory. `status` is not self-explanatory.\n3. **Two or three example values per categorical column.** This is the highest-value item and the most often omitted. A model that does not know `status` contains `shipped`, `pending`, `cancelled` will write `WHERE status = 'Shipped'` and get zero rows.\n4. **The joins that are allowed**, stated explicitly. Models invent plausible foreign keys.\n\nThis block is stable, so it belongs in the cached prefix. A 60-table schema is thousands of tokens on every request; select the tables relevant to the question first, using the same retrieval machinery you use for documents, and send five tables instead of sixty.\n\n## Door three: a tool call, and only the answer comes back\n\nWhere the question maps to a known operation (order status, account balance, inventory) do not generate SQL at all. Expose a function, let the model call it with parameters, and put only the result in the window.\n\nThis is safer (no generated SQL touching your database), cheaper (no schema in context), and more reliable (the query was written by you). It is the right default for the twenty questions that make up most of your traffic. Generated SQL is for the long tail you could not anticipate.\n\n## The failure to design against\n\nGenerated SQL fails silently. A query with a subtly wrong join returns rows (plausible rows, fewer than it should) and the model reports the number without any signal that anything went wrong. There is no exception, no empty result, nothing to catch.\n\nDefences, in order of effectiveness:\n\n- **Run it read-only**, as a role with `SELECT` on specific views and nothing else. Not \"instruct the model not to write\". A database permission.\n- **Return the query alongside the answer**, and show it in the interface. A user who knows their data spots a wrong filter instantly, and this is the single most effective check available.\n- **Assert on the shape.** Zero rows where rows were expected, a count larger than the table, a date outside the plausible range: cheap assertions that catch the loud failures.\n- **Cap and time-limit** every generated query. `LIMIT` and a statement timeout, enforced by the database.\n\n## Spreadsheets are not tables\n\nA worksheet a human maintains has merged cells, three header rows, a totals row in the middle, colour that carries meaning, and notes in column M. Reading it with `openpyxl` and serialising the cells produces confident nonsense.\n\nThe honest answer is that messy spreadsheets need a human-specified mapping (which rows are headers, which columns matter, what a blank means) and that this is per-file work. Automating it is a research problem, not an afternoon. Where it matters, build the importer for the specific file shape and validate it; where it does not, converting the sheet to a clean CSV by hand is a legitimate engineering decision rather than a defeat."},"check":{"question":"A text-to-SQL assistant reports plausible but wrong counts. The generated queries run without error. Which schema omission most directly causes this?","options":[{"text":"Column types were missing, so the model compared strings to numbers.","correct":false,"tempting":"Type mismatches usually raise database errors or return zero rows rather than plausible undercounts."},{"text":"Row counts per table were missing, so the model could not tell whether its result was reasonable.","correct":false,"tempting":"Would help a sanity check but does not cause the wrong query; the model is not verifying its own output either way."},{"text":"Example values for categorical columns were missing, so filters match a value the column never contains.","correct":true},{"text":"The schema was not in the cached prefix, so it was truncated on longer requests.","correct":false,"tempting":"Conflates a caching decision with a truncation policy; caching changes price, not whether content is present."}]},"prev":{"slug":"reranking-to-fit-the-budget","title":"Reranking, and why it reads better than the retriever","url":"https://addaly.com/learn/context-engineering/reranking-to-fit-the-budget"},"next":{"slug":"multi-hop-and-the-single-chunk-fallacy","title":"Questions no single passage answers","url":"https://addaly.com/learn/context-engineering/multi-hop-and-the-single-chunk-fallacy"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rows, records and schemas in the window', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/structured-sources-in-context (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rows, records and schemas in the window","courseTitle":"Context Engineering","module":{"index":3,"slug":"retrieval-as-a-pipeline","title":"Retrieval as a context pipeline"},"answer":null,"keyIdea":"Structured data reaches the window as records, as a schema the model writes a query against, or as the result of a function you wrote: choose by whether the answer is a lookup, an aggregate or a known operation, and never let generated SQL run with more than read permission on named views.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"not-everything-worth-retrieving-is-a-document","runs":[{"text":"Not everything worth retrieving is a document"}]},{"t":"p","runs":[{"text":"The question is “how many of my orders shipped late last month”. No amount of chunking and embedding will answer it, because the answer is a count over rows that no document contains. Somewhere between a third and a half of the questions people ask a business assistant are like this, and a pipeline built entirely around passage retrieval fails all of them in the same confident way."}]},{"t":"p","runs":[{"text":"Structured data enters the window through three different doors, and choosing the right one is most of the work."}]},{"t":"h2","id":"door-one-the-row-goes-in-the-context","runs":[{"text":"Door one: the row goes in the context"}]},{"t":"p","runs":[{"text":"Small, specific results. The customer’s order. Their current plan. Three open tickets."}]},{"t":"p","runs":[{"text":"Serialise them so each record stands alone:"}]},{"t":"code","lang":"","label":"Text","text":"order=ORD-88213 status=shipped placed=2026-08-14 delivered=2026-08-19\n  items=2 total=4,180 INR carrier=Bluedart"},{"t":"p","runs":[{"text":"Repeating the field names on every record costs tokens and buys robustness: the record survives being split, reordered or quoted in isolation. For a handful of records that trade is clearly right."}]},{"t":"p","runs":[{"text":"Above roughly twenty records it inverts, and a header-once format is much cheaper: the field names stop being worth their repetition. State the header immediately above the block so nothing can separate them."}]},{"t":"p","runs":[{"text":"Two rules regardless of format. "},{"text":"Include units and currency on every number","b":true},{"text":", because "},{"text":"4180","code":true},{"text":" and "},{"text":"4,180 INR","code":true},{"text":" are different facts and the model will guess. And "},{"text":"state the row count and whether the set is complete","b":true},{"text":": “12 of 87 orders, most recent first”. Without it the model treats twelve as all of them, which is the same partial-set failure the previous module described."}]},{"t":"h2","id":"door-two-the-schema-goes-in-the-context-and-the-model-writes","runs":[{"text":"Door two: the schema goes in the context, and the model writes a query"}]},{"t":"p","runs":[{"text":"For aggregates and joins (counts, sums, groupings) the data cannot go in the window and should not. Put the schema in and let the model produce SQL."}]},{"t":"p","runs":[{"text":"What the schema block needs, in order of value:"}]},{"t":"ol","start":1,"items":[[{"text":"Table and column names","b":true},{"text":", with types."}],[{"text":"A one-line description per column","b":true},{"text":" where the name is not self-explanatory. "},{"text":"status","code":true},{"text":" is not self-explanatory."}],[{"text":"Two or three example values per categorical column.","b":true},{"text":" This is the highest-value item and the most often omitted. A model that does not know "},{"text":"status","code":true},{"text":" contains "},{"text":"shipped","code":true},{"text":", "},{"text":"pending","code":true},{"text":", "},{"text":"cancelled","code":true},{"text":" will write "},{"text":"WHERE status = 'Shipped'","code":true},{"text":" and get zero rows."}],[{"text":"The joins that are allowed","b":true},{"text":", stated explicitly. Models invent plausible foreign keys."}]]},{"t":"p","runs":[{"text":"This block is stable, so it belongs in the cached prefix. A 60-table schema is thousands of tokens on every request; select the tables relevant to the question first, using the same retrieval machinery you use for documents, and send five tables instead of sixty."}]},{"t":"h2","id":"door-three-a-tool-call-and-only-the-answer-comes-back","runs":[{"text":"Door three: a tool call, and only the answer comes back"}]},{"t":"p","runs":[{"text":"Where the question maps to a known operation (order status, account balance, inventory) do not generate SQL at all. Expose a function, let the model call it with parameters, and put only the result in the window."}]},{"t":"p","runs":[{"text":"This is safer (no generated SQL touching your database), cheaper (no schema in context), and more reliable (the query was written by you). It is the right default for the twenty questions that make up most of your traffic. Generated SQL is for the long tail you could not anticipate."}]},{"t":"h2","id":"the-failure-to-design-against","runs":[{"text":"The failure to design against"}]},{"t":"p","runs":[{"text":"Generated SQL fails silently. A query with a subtly wrong join returns rows (plausible rows, fewer than it should) and the model reports the number without any signal that anything went wrong. There is no exception, no empty result, nothing to catch."}]},{"t":"p","runs":[{"text":"Defences, in order of effectiveness:"}]},{"t":"ul","items":[[{"text":"Run it read-only","b":true},{"text":", as a role with "},{"text":"SELECT","code":true},{"text":" on specific views and nothing else. Not “instruct the model not to write”. A database permission."}],[{"text":"Return the query alongside the answer","b":true},{"text":", and show it in the interface. A user who knows their data spots a wrong filter instantly, and this is the single most effective check available."}],[{"text":"Assert on the shape.","b":true},{"text":" Zero rows where rows were expected, a count larger than the table, a date outside the plausible range: cheap assertions that catch the loud failures."}],[{"text":"Cap and time-limit","b":true},{"text":" every generated query. "},{"text":"LIMIT","code":true},{"text":" and a statement timeout, enforced by the database."}]]},{"t":"h2","id":"spreadsheets-are-not-tables","runs":[{"text":"Spreadsheets are not tables"}]},{"t":"p","runs":[{"text":"A worksheet a human maintains has merged cells, three header rows, a totals row in the middle, colour that carries meaning, and notes in column M. Reading it with "},{"text":"openpyxl","code":true},{"text":" and serialising the cells produces confident nonsense."}]},{"t":"p","runs":[{"text":"The honest answer is that messy spreadsheets need a human-specified mapping (which rows are headers, which columns matter, what a blank means) and that this is per-file work. Automating it is a research problem, not an afternoon. Where it matters, build the importer for the specific file shape and validate it; where it does not, converting the sheet to a clean CSV by hand is a legitimate engineering decision rather than a defeat."}]}],"check":{"question":[{"text":"A text-to-SQL assistant reports plausible but wrong counts. The generated queries run without error. Which schema omission most directly causes this?"}],"options":[{"text":[{"text":"Column types were missing, so the model compared strings to numbers."}],"correct":false,"tempting":[{"text":"Type mismatches usually raise database errors or return zero rows rather than plausible undercounts."}]},{"text":[{"text":"Row counts per table were missing, so the model could not tell whether its result was reasonable."}],"correct":false,"tempting":[{"text":"Would help a sanity check but does not cause the wrong query; the model is not verifying its own output either way."}]},{"text":[{"text":"Example values for categorical columns were missing, so filters match a value the column never contains."}],"correct":true,"tempting":null},{"text":[{"text":"The schema was not in the cached prefix, so it was truncated on longer requests."}],"correct":false,"tempting":[{"text":"Conflates a caching decision with a truncation policy; caching changes price, not whether content is present."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"reranking-to-fit-the-budget","title":"Reranking, and why it reads better than the retriever","minutes":9},"next":{"slug":"multi-hop-and-the-single-chunk-fallacy","title":"Questions no single passage answers","minutes":8}}}