{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"plausible-but-wrong","title":"The new failure mode: plausible and wrong","url":"https://addaly.com/learn/how-software-changed/plausible-but-wrong","position":11,"of":87,"minutes":9,"answer":"Code that looks right is no longer evidence that it is right; polish and correctness came apart.","keyIdea":"Code that looks right is no longer evidence that it is right; polish and correctness came apart."},"body":{"format":"markdown","text":"## Human bugs and model bugs are shaped differently\n\nAfter a few years of reviewing people's work, you build a model of how humans get things wrong. Off-by-one at a boundary. Forgot the null case. Misread the ticket. Copy-pasted and changed one of the three places. Ran out of energy at the end of the function, so the error handling is thinner than the happy path.\n\nThat model no longer covers what you are reading. Generated code fails in a different distribution, and the difference matters more than the rate.\n\n## Three shapes worth recognising\n\n**The API that does not exist.** You get a call to a function or an argument that was never in the library:\n\n```python\ndf = df.drop_duplicates(subset=[\"invoice_id\"], keep=\"latest\")\n```\n\nThe real values are `\"first\"`, `\"last\"`, and `False`. This one is friendly, because it fails loudly:\n\n```\nValueError: keep must be either \"first\", \"last\" or False\n```\n\nYou find it the first time the line runs. The danger is only that it may not run until the monthly reconciliation job, at 02:00, in production.\n\n**The right shape with wrong semantics.** This one does not fail loudly:\n\n```python\n@retry(attempts=3, backoff=2.0)\ndef charge_customer(customer_id, amount_paise):\n    return payments.post(\"/v1/charges\", ...)\n```\n\nThat is textbook resilient engineering. It looks like something a thoughtful senior engineer wrote. It is also retrying a non-idempotent POST, so a network timeout after the charge succeeded will bill the customer twice. Nothing in the code looks wrong, because nothing in the code *is* wrong locally. It is wrong about the world.\n\nOthers in this family: timezone-naive datetimes in a system with users in Lagos and Auckland, float arithmetic on money, a `LIMIT 1000` quietly added to a query that a batch job depends on for completeness, a sort that is stable in the language it was pattern-matched from and unstable in yours.\n\n**The invented dependency.** A suggested import for a package that does not exist. If you install it without checking, you may find that someone else registered that name first. Attackers watch which package names models commonly invent and publish malware under them: the practice has picked up the name slopsquatting. Any `pip install`, `npm i`, or `go get` you were prompted into is a supply-chain decision, and it deserves ten seconds of looking at the actual registry page: who publishes it, since when, how many real dependents.\n\n## Why this defeats an old heuristic\n\nHere is the part to keep.\n\nExperienced reviewers use a fast, mostly unconscious check: *does this look like something a competent person wrote?* Sensible names, consistent structure, comments that match, sane error handling. For thirty years that was a decent proxy for correctness, because producing code that looked careful required someone to be careful.\n\nThat proxy is now broken. Appearance and correctness came apart. Polish is free and correctness is not.\n\n> Figure: Polish and correctness came apart. A two by two grid, rows Reads smoothly and Reads roughly, columns Correct and Wrong: Reads smoothly and Correct: Ship it; Reads smoothly and Wrong: The dangerous cell; Reads roughly and Correct: Tidy it up; Reads roughly and Wrong: Caught by instinct. For thirty years, code that looked careful was decent evidence that somebody had been careful, because producing that appearance took care. It is now free, and the proxy broke.\n\nSo the code that most deserves your suspicion is often the code that reads most smoothly: the confident, idiomatic, well-structured block that matches your house style exactly, because it was pattern-matched from a million examples of code that was right in a slightly different situation.\n\n## What to do instead\n\n**Verify every external call against the actual documentation.** Not from memory, and not by asking the same model. Open the docs. This costs about thirty seconds per unfamiliar call and catches the entire first category.\n\n**Compute one case by hand.** Take one realistic input, work out the expected output on paper, run it. This is old-fashioned and it catches semantic errors that no amount of reading catches, because reading is exactly the channel that has been compromised.\n\n**Ask the four boundary questions.** What does this do when the input is empty? When there are duplicates? When it arrives out of order? When it is a thousand times bigger than expected? Most silent semantic bugs live in one of those four.\n\n**Be most careful where the stakes are asymmetric.** Money, authentication, permissions, deletion, anything that sends a message to a human, anything that writes to another company's system. In those places, insist on understanding the code line by line, or do not ship it.\n\n## One reframe\n\nStop thinking of the output as a draft from a colleague and start thinking of it as a draft from a very well-read stranger who has never seen your system, cannot run it, will not be paged when it breaks, and will agree with you enthusiastically if you suggest something wrong. That framing gets your review posture roughly right."},"check":{"question":"You are reviewing a change and you see two blocks. Block A has an odd variable name, an inconsistent indent, and a comment that trails off. Block B is clean, idiomatic, well named, and matches the house style exactly. Both are new. Where should your careful attention go first?","options":[{"text":"Block A. Sloppiness in presentation usually indicates sloppiness in thinking, and that has not stopped being true.","correct":false,"tempting":"Relies on the appearance-to-care link, which still holds for human-written code but no longer holds for code where polish was free."},{"text":"Block B, because smooth, idiomatic code is now cheap to produce and carries no information about whether the logic suits your system.","correct":true},{"text":"Neither block specifically. Review the diff strictly in order so nothing is skipped and no bias creeps in.","correct":false,"tempting":"Fair as a completeness rule, but it spends equal attention everywhere, and attention is the scarce resource you are trying to allocate."},{"text":"Block A, since its rough edges suggest a human wrote it by hand and human bugs are the ones you know how to find.","correct":false,"tempting":"Inverts the priority: the bugs you already know how to find are precisely the ones that need less of your scarce attention."}]},"prev":{"slug":"reading-at-volume","title":"Reviewing code you did not write","url":"https://addaly.com/learn/how-software-changed/reading-at-volume"},"next":{"slug":"orienting-in-an-unfamiliar-system","title":"Orienting in a system you have never seen","url":"https://addaly.com/learn/how-software-changed/orienting-in-an-unfamiliar-system"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The new failure mode: plausible and wrong', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/plausible-but-wrong (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The new failure mode: plausible and wrong","courseTitle":"How Software Development Changed","module":{"index":2,"slug":"reading-what-you-did-not-write","title":"Reading what you did not write"},"answer":null,"keyIdea":"Code that looks right is no longer evidence that it is right; polish and correctness came apart.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Correct","Wrong"],"rows":["Reads smoothly","Reads roughly"],"type":"matrix","cells":[["Ship it","The dangerous cell"],["Tidy it up","Caught by instinct"]],"title":"Polish and correctness came apart","caption":"For thirty years, code that looked careful was decent evidence that somebody had been careful, because producing that appearance took care. It is now free, and the proxy broke.","cellLabels":[["what everybody wants","polish is free now, so this cell filled up"],["a real cost, and a visible one","the old reviewer check still fires here"]]},"alt":"Figure: Polish and correctness came apart. A two by two grid, rows Reads smoothly and Reads roughly, columns Correct and Wrong: Reads smoothly and Correct: Ship it; Reads smoothly and Wrong: The dangerous cell; Reads roughly and Correct: Tidy it up; Reads roughly and Wrong: Caught by instinct. For thirty years, code that looked careful was decent evidence that somebody had been careful, because producing that appearance took care. It is now free, and the proxy broke."}},"blocks":[{"t":"h2","id":"human-bugs-and-model-bugs-are-shaped-differently","runs":[{"text":"Human bugs and model bugs are shaped differently"}]},{"t":"p","runs":[{"text":"After a few years of reviewing people’s work, you build a model of how humans get things wrong. Off-by-one at a boundary. Forgot the null case. Misread the ticket. Copy-pasted and changed one of the three places. Ran out of energy at the end of the function, so the error handling is thinner than the happy path."}]},{"t":"p","runs":[{"text":"That model no longer covers what you are reading. Generated code fails in a different distribution, and the difference matters more than the rate."}]},{"t":"h2","id":"three-shapes-worth-recognising","runs":[{"text":"Three shapes worth recognising"}]},{"t":"p","runs":[{"text":"The API that does not exist.","b":true},{"text":" You get a call to a function or an argument that was never in the library:"}]},{"t":"code","lang":"python","label":"Python","text":"df = df.drop_duplicates(subset=[\"invoice_id\"], keep=\"latest\")"},{"t":"p","runs":[{"text":"The real values are "},{"text":"\"first\"","code":true},{"text":", "},{"text":"\"last\"","code":true},{"text":", and "},{"text":"False","code":true},{"text":". This one is friendly, because it fails loudly:"}]},{"t":"code","lang":"","label":"Text","text":"ValueError: keep must be either \"first\", \"last\" or False"},{"t":"p","runs":[{"text":"You find it the first time the line runs. The danger is only that it may not run until the monthly reconciliation job, at 02:00, in production."}]},{"t":"p","runs":[{"text":"The right shape with wrong semantics.","b":true},{"text":" This one does not fail loudly:"}]},{"t":"code","lang":"python","label":"Python","text":"@retry(attempts=3, backoff=2.0)\ndef charge_customer(customer_id, amount_paise):\n    return payments.post(\"/v1/charges\", ...)"},{"t":"p","runs":[{"text":"That is textbook resilient engineering. It looks like something a thoughtful senior engineer wrote. It is also retrying a non-idempotent POST, so a network timeout after the charge succeeded will bill the customer twice. Nothing in the code looks wrong, because nothing in the code "},{"text":"is","i":true},{"text":" wrong locally. It is wrong about the world."}]},{"t":"p","runs":[{"text":"Others in this family: timezone-naive datetimes in a system with users in Lagos and Auckland, float arithmetic on money, a "},{"text":"LIMIT 1000","code":true},{"text":" quietly added to a query that a batch job depends on for completeness, a sort that is stable in the language it was pattern-matched from and unstable in yours."}]},{"t":"p","runs":[{"text":"The invented dependency.","b":true},{"text":" A suggested import for a package that does not exist. If you install it without checking, you may find that someone else registered that name first. Attackers watch which package names models commonly invent and publish malware under them: the practice has picked up the name slopsquatting. Any "},{"text":"pip install","code":true},{"text":", "},{"text":"npm i","code":true},{"text":", or "},{"text":"go get","code":true},{"text":" you were prompted into is a supply-chain decision, and it deserves ten seconds of looking at the actual registry page: who publishes it, since when, how many real dependents."}]},{"t":"h2","id":"why-this-defeats-an-old-heuristic","runs":[{"text":"Why this defeats an old heuristic"}]},{"t":"p","runs":[{"text":"Here is the part to keep."}]},{"t":"p","runs":[{"text":"Experienced reviewers use a fast, mostly unconscious check: "},{"text":"does this look like something a competent person wrote?","i":true},{"text":" Sensible names, consistent structure, comments that match, sane error handling. For thirty years that was a decent proxy for correctness, because producing code that looked careful required someone to be careful."}]},{"t":"p","runs":[{"text":"That proxy is now broken. Appearance and correctness came apart. Polish is free and correctness is not."}]},{"t":"p","runs":[{"text":"So the code that most deserves your suspicion is often the code that reads most smoothly: the confident, idiomatic, well-structured block that matches your house style exactly, because it was pattern-matched from a million examples of code that was right in a slightly different situation."}]},{"t":"h2","id":"what-to-do-instead","runs":[{"text":"What to do instead"}]},{"t":"p","runs":[{"text":"Verify every external call against the actual documentation.","b":true},{"text":" Not from memory, and not by asking the same model. Open the docs. This costs about thirty seconds per unfamiliar call and catches the entire first category."}]},{"t":"p","runs":[{"text":"Compute one case by hand.","b":true},{"text":" Take one realistic input, work out the expected output on paper, run it. This is old-fashioned and it catches semantic errors that no amount of reading catches, because reading is exactly the channel that has been compromised."}]},{"t":"p","runs":[{"text":"Ask the four boundary questions.","b":true},{"text":" What does this do when the input is empty? When there are duplicates? When it arrives out of order? When it is a thousand times bigger than expected? Most silent semantic bugs live in one of those four."}]},{"t":"p","runs":[{"text":"Be most careful where the stakes are asymmetric.","b":true},{"text":" Money, authentication, permissions, deletion, anything that sends a message to a human, anything that writes to another company’s system. In those places, insist on understanding the code line by line, or do not ship it."}]},{"t":"h2","id":"one-reframe","runs":[{"text":"One reframe"}]},{"t":"p","runs":[{"text":"Stop thinking of the output as a draft from a colleague and start thinking of it as a draft from a very well-read stranger who has never seen your system, cannot run it, will not be paged when it breaks, and will agree with you enthusiastically if you suggest something wrong. That framing gets your review posture roughly right."}]}],"check":{"question":[{"text":"You are reviewing a change and you see two blocks. Block A has an odd variable name, an inconsistent indent, and a comment that trails off. Block B is clean, idiomatic, well named, and matches the house style exactly. Both are new. Where should your careful attention go first?"}],"options":[{"text":[{"text":"Block A. Sloppiness in presentation usually indicates sloppiness in thinking, and that has not stopped being true."}],"correct":false,"tempting":[{"text":"Relies on the appearance-to-care link, which still holds for human-written code but no longer holds for code where polish was free."}]},{"text":[{"text":"Block B, because smooth, idiomatic code is now cheap to produce and carries no information about whether the logic suits your system."}],"correct":true,"tempting":null},{"text":[{"text":"Neither block specifically. Review the diff strictly in order so nothing is skipped and no bias creeps in."}],"correct":false,"tempting":[{"text":"Fair as a completeness rule, but it spends equal attention everywhere, and attention is the scarce resource you are trying to allocate."}]},{"text":[{"text":"Block A, since its rough edges suggest a human wrote it by hand and human bugs are the ones you know how to find."}],"correct":false,"tempting":[{"text":"Inverts the priority: the bugs you already know how to find are precisely the ones that need less of your scarce attention."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"reading-at-volume","title":"Reviewing code you did not write","minutes":9},"next":{"slug":"orienting-in-an-unfamiliar-system","title":"Orienting in a system you have never seen","minutes":10}}}