{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"tasks-sorted-by-payoff","title":"Where the gain is large, small, and negative","url":"https://addaly.com/learn/how-software-changed/tasks-sorted-by-payoff","position":7,"of":87,"minutes":10,"answer":"The payoff from generated code tracks how cheap verification is, not how easy the code is.","keyIdea":"The payoff from generated code tracks how cheap verification is, not how easy the code is, so the strongest use is work with a perfect oracle such as a refactor and the weakest is work whose difficulty was deciding what should happen."},"body":{"format":"markdown","text":"## One formula, then the list\n\nThe value of using a generator on a given task is roughly:\n\n```\nvalue = time_saved_generating\n      − time_spent_verifying\n      − P(defect) × cost_of_that_defect\n```\n\nMost arguments about whether these tools help are people quoting different terms of that expression at each other. The first term is easy to see and easy to feel. The second is measurable if you bother. The third is the one that decides the answer and the one nobody estimates.\n\nThe practical consequence: **the payoff is high exactly where verification is cheap.** Not where the code is easy. Where checking is easy. Those are different properties and confusing them is the single most common mistake.\n\n> Figure: Sorted by how cheap the check is. Cheap to verify, so the payoff is large: A mechanical refactor, where the old version is a perfect oracle; Format translation you check by running it once; A language you do not know, in a domain you do; Boilerplate whose failure is loud and immediate; Explaining a diff, a dense regex or a stack trace. Expensive to verify, so the payoff is small or negative: Pricing rules, permission models, tax logic: the difficulty was deciding; Concurrency, where the failure is intermittent and load-dependent; Deep changes in a system you already hold in your head; Cryptographic primitives, where the rule is still to use the vetted library; Anything whose answer you will only see in production. The dividing line is not how hard the code is. It is how cheaply you can tell whether it is right, and those are different properties.\n\n## Large payoff\n\n- **A language or framework you do not know, in a domain you do.** Writing a Terraform module when you understand networking but not HCL. You can read the plan output and tell whether it is right; you just could not have typed it.\n- **Format translation with an obvious oracle.** A curl command into Python, an OpenAPI spec into typed clients, a JSON sample into a schema, a CSV header into a table definition. You check by running it once.\n- **Boilerplate whose failure is loud.** Serialisers, fixtures, argument parsing, a parser for a documented format. Wrong output is visibly wrong.\n- **One-off scripts whose output you will read.** A migration check, a log analysis, a data reshape. The result is the verification.\n- **Explaining things.** An unfamiliar diff, a dense regex, a stack trace, a compiler error you have never seen. Low risk, because you can verify the explanation against the artefact in front of you.\n- **Mechanical refactors.** The oracle is perfect: behave identically, and the old version is in git. This is the single best-suited category of work, and it is undersold.\n- **Test scaffolding.** Parametrisation, fixtures, mock setup. Not the assertions: the assertions are the oracle and must come from the requirement.\n\n## Medium payoff\n\n- **Well-scoped features in a stack you know.** Real gains on the first draft, then you pay the full review cost, and the review cost is a large fraction of the total.\n- **Documentation drafts.** Fast to produce, and you will rewrite half, because the parts a reader needs are the parts only you know.\n- **SQL.** Genuinely useful for getting the shape of a query, and you must read the execution plan, because a query that returns the right rows and scans the whole table is a production incident waiting for the data to grow.\n\n## Small or negative payoff\n\n- **Deep changes in a system you know well.** Your context is larger than anything you can fit in a prompt, and the cost of transferring it exceeds the cost of typing. This is the setting where the careful trial found a negative effect, and the mechanism is exactly this.\n- **Work whose difficulty is the specification.** Pricing rules, permission models, workflow state machines, tax logic. A confident answer to an under-specified question is worse than no answer, because it looks like a decision.\n- **Concurrency.** Locks, memory ordering, async cancellation. Plausible-looking concurrency code is the most dangerous artefact in this entire list, because the failure is intermittent, load-dependent, and will not appear in review or in your test suite.\n- **Novel algorithms and performance work.** Correctness is subtle, the reference implementation does not exist, and the failure is a wrong number rather than an exception.\n- **Cryptographic primitives.** The rule was always *do not write your own*, and it got more important, not less. Use the vetted library. The generated implementation of AES-GCM that passes your test vectors may still leak through timing.\n- **Anything where you cannot cheaply tell whether it is right.** The general case of all of the above.\n\n## The test to apply before you start\n\nBefore generating anything, answer one question out loud: **how will I know this is correct, and how long will that take?**\n\nIf the answer is \"run it and look at the output\": proceed, the payoff is probably large. If it is \"read it carefully and reason about the edge cases\": the payoff is modest, and you should ask for something smaller. If it is \"we will find out in production\": do not generate it; that is a decision, and decisions are the part of the job that did not get cheaper.\n\n## A note on the free path\n\nNone of the categories above depend on which tool you use. A quantised open-weight coding model running locally through Ollama handles the entire *large payoff* list adequately, because those tasks have cheap oracles and low stakes. The gap between a local model and a hosted frontier one shows up in the medium and negative categories (multi-file reasoning, long context, subtle semantics) which is precisely the work you should be doing yourself anyway. That is a convenient alignment and it is worth knowing before you spend money."},"check":{"question":"Two tasks: rewriting a 400-line module to use async I/O with the existing test suite as the reference, and implementing a new discount rule described in a two-line ticket. Both are about the same size. Which is the better candidate for generation and why?","options":[{"text":"The discount rule, because it is new code with no existing behaviour to preserve, so there is less risk of breaking something that already works.","correct":false,"tempting":"Confuses absence of existing behaviour with absence of risk; with no reference behaviour there is also no oracle, which is what makes it the harder case."},{"text":"The async rewrite, because concurrency is a well-documented pattern that appears constantly in training data and is therefore reliably generated.","correct":false,"tempting":"Right answer for the wrong reason, and the reason is dangerous: familiarity in training data says nothing about whether this system's concurrency is correct."},{"text":"The async rewrite, because the required behaviour is already defined and executable, so checking the result is cheap.","correct":true},{"text":"Neither, because both involve changing production behaviour and generation should be limited to scaffolding, scripts and documentation until a team has verification tooling in place.","correct":false,"tempting":"Over-generalises a real caution into a rule that gives up the highest-payoff category, mechanical change against an existing oracle."}]},"prev":{"slug":"queues-not-speeds","title":"Queues, not speeds","url":"https://addaly.com/learn/how-software-changed/queues-not-speeds"},"next":{"slug":"what-the-loop-costs","title":"What the loop costs, and the free path","url":"https://addaly.com/learn/how-software-changed/what-the-loop-costs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the gain is large, small, and negative', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/tasks-sorted-by-payoff (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the gain is large, small, and negative","courseTitle":"How Software Development Changed","module":{"index":1,"slug":"where-the-cost-went","title":"Where the cost actually went"},"answer":null,"keyIdea":"The payoff from generated code tracks how cheap verification is, not how easy the code is, so the strongest use is work with a perfect oracle such as a refactor and the weakest is work whose difficulty was deciding what should happen.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Cheap to verify, so the payoff is large","items":["A mechanical refactor, where the old version is a perfect oracle","Format translation you check by running it once","A language you do not know, in a domain you do","Boilerplate whose failure is loud and immediate","Explaining a diff, a dense regex or a stack trace"]},"type":"compare","right":{"title":"Expensive to verify, so the payoff is small or negative","items":["Pricing rules, permission models, tax logic: the difficulty was deciding","Concurrency, where the failure is intermittent and load-dependent","Deep changes in a system you already hold in your head","Cryptographic primitives, where the rule is still to use the vetted library","Anything whose answer you will only see in production"]},"title":"Sorted by how cheap the check is","caption":"The dividing line is not how hard the code is. It is how cheaply you can tell whether it is right, and those are different properties."},"alt":"Figure: Sorted by how cheap the check is. Cheap to verify, so the payoff is large: A mechanical refactor, where the old version is a perfect oracle; Format translation you check by running it once; A language you do not know, in a domain you do; Boilerplate whose failure is loud and immediate; Explaining a diff, a dense regex or a stack trace. Expensive to verify, so the payoff is small or negative: Pricing rules, permission models, tax logic: the difficulty was deciding; Concurrency, where the failure is intermittent and load-dependent; Deep changes in a system you already hold in your head; Cryptographic primitives, where the rule is still to use the vetted library; Anything whose answer you will only see in production. The dividing line is not how hard the code is. It is how cheaply you can tell whether it is right, and those are different properties."}},"blocks":[{"t":"h2","id":"one-formula-then-the-list","runs":[{"text":"One formula, then the list"}]},{"t":"p","runs":[{"text":"The value of using a generator on a given task is roughly:"}]},{"t":"code","lang":"","label":"Text","text":"value = time_saved_generating\n      − time_spent_verifying\n      − P(defect) × cost_of_that_defect"},{"t":"p","runs":[{"text":"Most arguments about whether these tools help are people quoting different terms of that expression at each other. The first term is easy to see and easy to feel. The second is measurable if you bother. The third is the one that decides the answer and the one nobody estimates."}]},{"t":"p","runs":[{"text":"The practical consequence: "},{"text":"the payoff is high exactly where verification is cheap.","b":true},{"text":" Not where the code is easy. Where checking is easy. Those are different properties and confusing them is the single most common mistake."}]},{"t":"h2","id":"large-payoff","runs":[{"text":"Large payoff"}]},{"t":"ul","items":[[{"text":"A language or framework you do not know, in a domain you do.","b":true},{"text":" Writing a Terraform module when you understand networking but not HCL. You can read the plan output and tell whether it is right; you just could not have typed it."}],[{"text":"Format translation with an obvious oracle.","b":true},{"text":" A curl command into Python, an OpenAPI spec into typed clients, a JSON sample into a schema, a CSV header into a table definition. You check by running it once."}],[{"text":"Boilerplate whose failure is loud.","b":true},{"text":" Serialisers, fixtures, argument parsing, a parser for a documented format. Wrong output is visibly wrong."}],[{"text":"One-off scripts whose output you will read.","b":true},{"text":" A migration check, a log analysis, a data reshape. The result is the verification."}],[{"text":"Explaining things.","b":true},{"text":" An unfamiliar diff, a dense regex, a stack trace, a compiler error you have never seen. Low risk, because you can verify the explanation against the artefact in front of you."}],[{"text":"Mechanical refactors.","b":true},{"text":" The oracle is perfect: behave identically, and the old version is in git. This is the single best-suited category of work, and it is undersold."}],[{"text":"Test scaffolding.","b":true},{"text":" Parametrisation, fixtures, mock setup. Not the assertions: the assertions are the oracle and must come from the requirement."}]]},{"t":"h2","id":"medium-payoff","runs":[{"text":"Medium payoff"}]},{"t":"ul","items":[[{"text":"Well-scoped features in a stack you know.","b":true},{"text":" Real gains on the first draft, then you pay the full review cost, and the review cost is a large fraction of the total."}],[{"text":"Documentation drafts.","b":true},{"text":" Fast to produce, and you will rewrite half, because the parts a reader needs are the parts only you know."}],[{"text":"SQL.","b":true},{"text":" Genuinely useful for getting the shape of a query, and you must read the execution plan, because a query that returns the right rows and scans the whole table is a production incident waiting for the data to grow."}]]},{"t":"h2","id":"small-or-negative-payoff","runs":[{"text":"Small or negative payoff"}]},{"t":"ul","items":[[{"text":"Deep changes in a system you know well.","b":true},{"text":" Your context is larger than anything you can fit in a prompt, and the cost of transferring it exceeds the cost of typing. This is the setting where the careful trial found a negative effect, and the mechanism is exactly this."}],[{"text":"Work whose difficulty is the specification.","b":true},{"text":" Pricing rules, permission models, workflow state machines, tax logic. A confident answer to an under-specified question is worse than no answer, because it looks like a decision."}],[{"text":"Concurrency.","b":true},{"text":" Locks, memory ordering, async cancellation. Plausible-looking concurrency code is the most dangerous artefact in this entire list, because the failure is intermittent, load-dependent, and will not appear in review or in your test suite."}],[{"text":"Novel algorithms and performance work.","b":true},{"text":" Correctness is subtle, the reference implementation does not exist, and the failure is a wrong number rather than an exception."}],[{"text":"Cryptographic primitives.","b":true},{"text":" The rule was always "},{"text":"do not write your own","i":true},{"text":", and it got more important, not less. Use the vetted library. The generated implementation of AES-GCM that passes your test vectors may still leak through timing."}],[{"text":"Anything where you cannot cheaply tell whether it is right.","b":true},{"text":" The general case of all of the above."}]]},{"t":"h2","id":"the-test-to-apply-before-you-start","runs":[{"text":"The test to apply before you start"}]},{"t":"p","runs":[{"text":"Before generating anything, answer one question out loud: "},{"text":"how will I know this is correct, and how long will that take?","b":true}]},{"t":"p","runs":[{"text":"If the answer is “run it and look at the output”: proceed, the payoff is probably large. If it is “read it carefully and reason about the edge cases”: the payoff is modest, and you should ask for something smaller. If it is “we will find out in production”: do not generate it; that is a decision, and decisions are the part of the job that did not get cheaper."}]},{"t":"h2","id":"a-note-on-the-free-path","runs":[{"text":"A note on the free path"}]},{"t":"p","runs":[{"text":"None of the categories above depend on which tool you use. A quantised open-weight coding model running locally through Ollama handles the entire "},{"text":"large payoff","i":true},{"text":" list adequately, because those tasks have cheap oracles and low stakes. The gap between a local model and a hosted frontier one shows up in the medium and negative categories (multi-file reasoning, long context, subtle semantics) which is precisely the work you should be doing yourself anyway. That is a convenient alignment and it is worth knowing before you spend money."}]}],"check":{"question":[{"text":"Two tasks: rewriting a 400-line module to use async I/O with the existing test suite as the reference, and implementing a new discount rule described in a two-line ticket. Both are about the same size. Which is the better candidate for generation and why?"}],"options":[{"text":[{"text":"The discount rule, because it is new code with no existing behaviour to preserve, so there is less risk of breaking something that already works."}],"correct":false,"tempting":[{"text":"Confuses absence of existing behaviour with absence of risk; with no reference behaviour there is also no oracle, which is what makes it the harder case."}]},{"text":[{"text":"The async rewrite, because concurrency is a well-documented pattern that appears constantly in training data and is therefore reliably generated."}],"correct":false,"tempting":[{"text":"Right answer for the wrong reason, and the reason is dangerous: familiarity in training data says nothing about whether this system’s concurrency is correct."}]},{"text":[{"text":"The async rewrite, because the required behaviour is already defined and executable, so checking the result is cheap."}],"correct":true,"tempting":null},{"text":[{"text":"Neither, because both involve changing production behaviour and generation should be limited to scaffolding, scripts and documentation until a team has verification tooling in place."}],"correct":false,"tempting":[{"text":"Over-generalises a real caution into a rule that gives up the highest-payoff category, mechanical change against an existing oracle."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"queues-not-speeds","title":"Queues, not speeds","minutes":10},"next":{"slug":"what-the-loop-costs","title":"What the loop costs, and the free path","minutes":9}}}