Where the gain is large, small, and negative

One formula, then the list

The value of using a generator on a given task is roughly:

value = time_saved_generating
      − time_spent_verifying
      − P(defect) × cost_of_that_defect

Most arguments about whether these tools help are people quoting different terms of that expression at each other. The first term is easy to see and easy to feel. The second is measurable if you bother. The third is the one that decides the answer and the one nobody estimates.

The practical consequence: the payoff is high exactly where verification is cheap. Not where the code is easy. Where checking is easy. Those are different properties and confusing them is the single most common mistake.

Large payoff

• A language or framework you do not know, in a domain you do. Writing a Terraform module when you understand networking but not HCL. You can read the plan output and tell whether it is right; you just could not have typed it.
• Format translation with an obvious oracle. A curl command into Python, an OpenAPI spec into typed clients, a JSON sample into a schema, a CSV header into a table definition. You check by running it once.
• Boilerplate whose failure is loud. Serialisers, fixtures, argument parsing, a parser for a documented format. Wrong output is visibly wrong.
• One-off scripts whose output you will read. A migration check, a log analysis, a data reshape. The result is the verification.
• Explaining things. An unfamiliar diff, a dense regex, a stack trace, a compiler error you have never seen. Low risk, because you can verify the explanation against the artefact in front of you.
• Mechanical refactors. The oracle is perfect: behave identically, and the old version is in git. This is the single best-suited category of work, and it is undersold.
• Test scaffolding. Parametrisation, fixtures, mock setup. Not the assertions: the assertions are the oracle and must come from the requirement.

Medium payoff

• Well-scoped features in a stack you know. Real gains on the first draft, then you pay the full review cost, and the review cost is a large fraction of the total.
• Documentation drafts. Fast to produce, and you will rewrite half, because the parts a reader needs are the parts only you know.
• SQL. Genuinely useful for getting the shape of a query, and you must read the execution plan, because a query that returns the right rows and scans the whole table is a production incident waiting for the data to grow.

Small or negative payoff

• Deep changes in a system you know well. Your context is larger than anything you can fit in a prompt, and the cost of transferring it exceeds the cost of typing. This is the setting where the careful trial found a negative effect, and the mechanism is exactly this.
• Work whose difficulty is the specification. Pricing rules, permission models, workflow state machines, tax logic. A confident answer to an under-specified question is worse than no answer, because it looks like a decision.
• Concurrency. Locks, memory ordering, async cancellation. Plausible-looking concurrency code is the most dangerous artefact in this entire list, because the failure is intermittent, load-dependent, and will not appear in review or in your test suite.
• Novel algorithms and performance work. Correctness is subtle, the reference implementation does not exist, and the failure is a wrong number rather than an exception.
• Cryptographic primitives. The rule was always do not write your own, and it got more important, not less. Use the vetted library. The generated implementation of AES-GCM that passes your test vectors may still leak through timing.
• Anything where you cannot cheaply tell whether it is right. The general case of all of the above.

The test to apply before you start

Before generating anything, answer one question out loud: how will I know this is correct, and how long will that take?

If the answer is “run it and look at the output”: proceed, the payoff is probably large. If it is “read it carefully and reason about the edge cases”: the payoff is modest, and you should ask for something smaller. If it is “we will find out in production”: do not generate it; that is a decision, and decisions are the part of the job that did not get cheaper.

A note on the free path

None of the categories above depend on which tool you use. A quantised open-weight coding model running locally through Ollama handles the entire large payoff list adequately, because those tasks have cheap oracles and low stakes. The gap between a local model and a hosted frontier one shows up in the medium and negative categories (multi-file reasoning, long context, subtle semantics) which is precisely the work you should be doing yourself anyway. That is a convenient alignment and it is worth knowing before you spend money.