{"course":{"slug":"prompting","title":"Getting Real Work out of a Model","url":"https://addaly.com/learn/prompting","level":"beginner","track":"start"},"lesson":{"slug":"output-that-fails-visibly","title":"Building the failure into the format","url":"https://addaly.com/learn/prompting/output-that-fails-visibly","position":36,"of":72,"minutes":8,"answer":"Give the output a legal way to fail (a stated empty value, a source quote, a confidence field, a permitted refusal) so the rows that need checking identify themselves.","keyIdea":"Give the output a legal way to fail (a stated empty value, a source quote, a confidence field, a permitted refusal) so the rows that need checking identify themselves."},"body":{"format":"markdown","text":"## A blank is better than a guess\n\nThe most expensive output failure is not an error message. It is a well-formed answer containing something the model made up, sitting in a column beside forty things it did not.\n\nYou cannot prevent invention with a prohibition, as you know. What you can do is build a legal way to fail into the shape of the output, so that failing is easier than inventing.\n\nFour devices, all cheap.\n\n## 1. A stated empty value\n\nEvery field gets a legal absence.\n\n> Where the document does not state a value, write `NOT_STATED`. Do not infer, estimate, or use a typical value.\n\nThe words matter. \"Do not infer\" blocks the reasonable-sounding deduction. \"Do not use a typical value\" blocks the most common failure of all, which is a standard figure appearing where a specific one was missing.\n\nThen you can find them: sort by that column and every gap is visible in one glance. A gap you can see is a gap you can go and fill from the original.\n\n## 2. A source column\n\nFor anything extracted from a document, ask where it came from.\n\n> Columns: field, value, and the exact quoted sentence it comes from. If you cannot quote a sentence, the value is NOT_STATED.\n\nThis is the strongest single technique in this course for document work, and the reason is structural rather than motivational: a quote is checkable in four seconds with a search, and a value that has no quotable source cannot be produced without the absence being visible.\n\nYou will also find that requiring a quote reduces invention, because there is nowhere for an unsupported value to sit.\n\n## 3. A flag rather than a hedge\n\nHedging inside prose is invisible at scale. \"This may be approximately 40,000\" reads as 40,000 by the third row.\n\nGive uncertainty a field of its own:\n\n> Add a column `confidence` with one of: `stated` (explicit in the document), `derived` (calculated from stated figures; show the calculation), `uncertain` (interpretation required; say why).\n\nThree values, mechanically defined. That is much more useful than a percentage, which a model will produce and which is not calibrated to anything. You can filter on `uncertain` and check only those rows.\n\n## 4. A refusal that is a valid answer\n\nGive \"this cannot be done\" a shape, so it is not a conversational failure:\n\n> If the question cannot be answered from the material provided, reply with exactly: `CANNOT ANSWER: ` followed by what is missing.\n\nWithout a permitted shape for failure, the model produces an answer, because producing an answer is what the exchange calls for. With one, \"cannot answer\" competes on equal terms.\n\nThe second half, *what is missing*, turns a dead end into your next action. Nine times in ten it names a document you have and did not paste.\n\n## What this looks like put together\n\n```\nFor each of the 12 policies, output:\npolicy_id | premium | excess | maternity_covered | source_quote | confidence\n\nRules:\n- source_quote: the exact sentence the value came from, or empty.\n- If there is no sentence to quote, the value is NOT_STATED.\n- confidence: stated / derived / uncertain.\n- If a whole policy document is missing or unreadable, output the row\n  as CANNOT READ and nothing else for that policy.\n```\n\nNow three failure modes are visible without reading anything: missing values are `NOT_STATED`, shaky values are `uncertain`, missing documents are `CANNOT READ`. What is left is checkable in the time it takes to scan a column.\n\n## Why this beats checking harder\n\nYou cannot verify forty rows carefully. Nobody does, whatever they say at the start.\n\nWhat you can do is make the rows that need attention identify themselves, and then check those properly. That is a format decision, made before you send the prompt, and it is the difference between a batch you can trust and a batch you have merely received.\n\n## The honest limit\n\nNone of this makes the values right. A quoted sentence can be quoted accurately and read wrongly. A row marked `stated` can point at the wrong stated figure: the subtotal rather than the total, the order date rather than the invoice date.\n\nWhat these devices buy is *visibility*, which is a smaller claim and a more useful one. They separate the rows where something is missing or shaky from the rows where something is asserted with a source, and they hand you a five-minute check instead of an unperformable forty-minute one. The remaining errors are the ones that look exactly like correct answers, and those are the subject of a later module."},"check":{"question":"Why does requiring a quoted source sentence for every extracted value reduce fabrication more effectively than instructing the model not to fabricate?","options":[{"text":"Because quoting is a simpler task than extraction, so accuracy improves across the board.","correct":false,"tempting":"Quoting is not simpler in the relevant sense, and the effect is about visibility rather than general accuracy."},{"text":"Because a value with no quotable sentence has nowhere to sit in the output, and any quote that is offered can be checked against the document in seconds.","correct":true},{"text":"Because the instruction to quote is longer, and longer instructions are followed more reliably.","correct":false,"tempting":"Length is not what makes an instruction hold; the mechanism is that the output now carries checkable evidence."},{"text":"Because models are trained to be accurate when asked to cite sources.","correct":false,"tempting":"Invents a training property; citation behaviour is not a guarantee, as invented citations demonstrate."}]},"prev":{"slug":"asking-in-one-language","title":"Working across languages","url":"https://addaly.com/learn/prompting/asking-in-one-language"},"next":{"slug":"make-it-show-its-work","title":"Make it show its work","url":"https://addaly.com/learn/prompting/make-it-show-its-work"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Building the failure into the format', in Getting Real Work out of a Model. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/prompting/output-that-fails-visibly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Building the failure into the format","courseTitle":"Getting Real Work out of a Model","module":{"index":4,"slug":"the-shape-of-the-answer","title":"The shape of the answer"},"answer":null,"keyIdea":"Give the output a legal way to fail (a stated empty value, a source quote, a confidence field, a permitted refusal) so the rows that need checking identify themselves.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-blank-is-better-than-a-guess","runs":[{"text":"A blank is better than a guess"}]},{"t":"p","runs":[{"text":"The most expensive output failure is not an error message. It is a well-formed answer containing something the model made up, sitting in a column beside forty things it did not."}]},{"t":"p","runs":[{"text":"You cannot prevent invention with a prohibition, as you know. What you can do is build a legal way to fail into the shape of the output, so that failing is easier than inventing."}]},{"t":"p","runs":[{"text":"Four devices, all cheap."}]},{"t":"h2","id":"1-a-stated-empty-value","runs":[{"text":"1. A stated empty value"}]},{"t":"p","runs":[{"text":"Every field gets a legal absence."}]},{"t":"quote","runs":[{"text":"Where the document does not state a value, write "},{"text":"NOT_STATED","code":true},{"text":". Do not infer, estimate, or use a typical value."}]},{"t":"p","runs":[{"text":"The words matter. “Do not infer” blocks the reasonable-sounding deduction. “Do not use a typical value” blocks the most common failure of all, which is a standard figure appearing where a specific one was missing."}]},{"t":"p","runs":[{"text":"Then you can find them: sort by that column and every gap is visible in one glance. A gap you can see is a gap you can go and fill from the original."}]},{"t":"h2","id":"2-a-source-column","runs":[{"text":"2. A source column"}]},{"t":"p","runs":[{"text":"For anything extracted from a document, ask where it came from."}]},{"t":"quote","runs":[{"text":"Columns: field, value, and the exact quoted sentence it comes from. If you cannot quote a sentence, the value is NOT_STATED."}]},{"t":"p","runs":[{"text":"This is the strongest single technique in this course for document work, and the reason is structural rather than motivational: a quote is checkable in four seconds with a search, and a value that has no quotable source cannot be produced without the absence being visible."}]},{"t":"p","runs":[{"text":"You will also find that requiring a quote reduces invention, because there is nowhere for an unsupported value to sit."}]},{"t":"h2","id":"3-a-flag-rather-than-a-hedge","runs":[{"text":"3. A flag rather than a hedge"}]},{"t":"p","runs":[{"text":"Hedging inside prose is invisible at scale. “This may be approximately 40,000” reads as 40,000 by the third row."}]},{"t":"p","runs":[{"text":"Give uncertainty a field of its own:"}]},{"t":"quote","runs":[{"text":"Add a column "},{"text":"confidence","code":true},{"text":" with one of: "},{"text":"stated","code":true},{"text":" (explicit in the document), "},{"text":"derived","code":true},{"text":" (calculated from stated figures; show the calculation), "},{"text":"uncertain","code":true},{"text":" (interpretation required; say why)."}]},{"t":"p","runs":[{"text":"Three values, mechanically defined. That is much more useful than a percentage, which a model will produce and which is not calibrated to anything. You can filter on "},{"text":"uncertain","code":true},{"text":" and check only those rows."}]},{"t":"h2","id":"4-a-refusal-that-is-a-valid-answer","runs":[{"text":"4. A refusal that is a valid answer"}]},{"t":"p","runs":[{"text":"Give “this cannot be done” a shape, so it is not a conversational failure:"}]},{"t":"quote","runs":[{"text":"If the question cannot be answered from the material provided, reply with exactly: "},{"text":"CANNOT ANSWER: ","code":true},{"text":" followed by what is missing."}]},{"t":"p","runs":[{"text":"Without a permitted shape for failure, the model produces an answer, because producing an answer is what the exchange calls for. With one, “cannot answer” competes on equal terms."}]},{"t":"p","runs":[{"text":"The second half, "},{"text":"what is missing","i":true},{"text":", turns a dead end into your next action. Nine times in ten it names a document you have and did not paste."}]},{"t":"h2","id":"what-this-looks-like-put-together","runs":[{"text":"What this looks like put together"}]},{"t":"code","lang":"","label":"Text","text":"For each of the 12 policies, output:\npolicy_id | premium | excess | maternity_covered | source_quote | confidence\n\nRules:\n- source_quote: the exact sentence the value came from, or empty.\n- If there is no sentence to quote, the value is NOT_STATED.\n- confidence: stated / derived / uncertain.\n- If a whole policy document is missing or unreadable, output the row\n  as CANNOT READ and nothing else for that policy."},{"t":"p","runs":[{"text":"Now three failure modes are visible without reading anything: missing values are "},{"text":"NOT_STATED","code":true},{"text":", shaky values are "},{"text":"uncertain","code":true},{"text":", missing documents are "},{"text":"CANNOT READ","code":true},{"text":". What is left is checkable in the time it takes to scan a column."}]},{"t":"h2","id":"why-this-beats-checking-harder","runs":[{"text":"Why this beats checking harder"}]},{"t":"p","runs":[{"text":"You cannot verify forty rows carefully. Nobody does, whatever they say at the start."}]},{"t":"p","runs":[{"text":"What you can do is make the rows that need attention identify themselves, and then check those properly. That is a format decision, made before you send the prompt, and it is the difference between a batch you can trust and a batch you have merely received."}]},{"t":"h2","id":"the-honest-limit","runs":[{"text":"The honest limit"}],"role":"limit"},{"t":"p","runs":[{"text":"None of this makes the values right. A quoted sentence can be quoted accurately and read wrongly. A row marked "},{"text":"stated","code":true},{"text":" can point at the wrong stated figure: the subtotal rather than the total, the order date rather than the invoice date."}]},{"t":"p","runs":[{"text":"What these devices buy is "},{"text":"visibility","i":true},{"text":", which is a smaller claim and a more useful one. They separate the rows where something is missing or shaky from the rows where something is asserted with a source, and they hand you a five-minute check instead of an unperformable forty-minute one. The remaining errors are the ones that look exactly like correct answers, and those are the subject of a later module."}]}],"check":{"question":[{"text":"Why does requiring a quoted source sentence for every extracted value reduce fabrication more effectively than instructing the model not to fabricate?"}],"options":[{"text":[{"text":"Because quoting is a simpler task than extraction, so accuracy improves across the board."}],"correct":false,"tempting":[{"text":"Quoting is not simpler in the relevant sense, and the effect is about visibility rather than general accuracy."}]},{"text":[{"text":"Because a value with no quotable sentence has nowhere to sit in the output, and any quote that is offered can be checked against the document in seconds."}],"correct":true,"tempting":null},{"text":[{"text":"Because the instruction to quote is longer, and longer instructions are followed more reliably."}],"correct":false,"tempting":[{"text":"Length is not what makes an instruction hold; the mechanism is that the output now carries checkable evidence."}]},{"text":[{"text":"Because models are trained to be accurate when asked to cite sources."}],"correct":false,"tempting":[{"text":"Invents a training property; citation behaviour is not a guarantee, as invented citations demonstrate."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"asking-in-one-language","title":"Working across languages","minutes":9},"next":{"slug":"make-it-show-its-work","title":"Make it show its work","minutes":8}}}