Where the cost actually went

9 lessons · 1 h 22

You will be able toAccount for a productivity claim in numbers: name which stage of the delivery pipeline got faster, compute what that does to end-to-end throughput when the constraint sits elsewhere, and read a study or a vendor figure well enough to say precisely what it does and does not establish.

An illustrative case, built from documented patternsThe quarter the pull requests doubledA 34-person product team at a logistics company in Pune, six months after every engineer got an assistant licence

Meera runs delivery for a team that builds routing and proof-of-delivery software for a fleet operator. In January the company bought assistant licences for all 34 engineers. By April the dashboards looked excellent.

Commits were up 78 per cent on the same quarter last year. Pull requests opened had gone from an average of 21 a week to 41. In the all-hands, the CTO used the phrase “step change in output” and nobody in the room disagreed, because everybody felt it.

Then the head of the fleet operator’s operations team called to ask why a feature he had signed off in February was still not live.

Meera pulled four numbers out of the git host and the deploy log, by hand, into a spreadsheet. It took an afternoon.

Deployments per week: 9 in the previous year, 9 now.
Median time from first commit to production: 1.3 weeks then, 2.9 weeks now.
Open pull requests at any moment: 25 then, 46 now.
Changes that came back within thirty days as a revert, a hotfix or an incident: 11 per cent then, 19 per cent now.

The team was producing twice as much and delivering the same amount, more slowly, with worse outcomes. Nothing in the pipeline had changed except the rate at which work arrived at the review stage. Reviewers were the same eleven people with the same days, and they had started skimming, which is what a queue does when it cannot grow the service rate: it lowers the service quality until arrival and service match again.

Meera worked out the arithmetic she would need. Work in progress equals arrival rate times time in system, so with 46 open and about 20 merging a week, an item waited about 2.3 weeks. Capping the team at 12 in flight would bring that to about 0.6 weeks at the same throughput.

That left her with a decision, and both sides of it cost something real.

Cap work in progress. Each engineer limited to two open pull requests. On hitting the cap you do not open a third; you go and review, or you go and get one of yours merged. The mechanism is automatic and it moves effort to the constraint without anybody having to be persuaded each time.

The cost: every activity number on the executive dashboard falls. Commits down, pull requests down, and Meera would have to stand in a quarterly review and say that her team had deliberately reduced its output in the same quarter the company paid for tooling meant to increase it. Two of her engineers were up for promotion and their cases had been written around shipped volume. The finance director had already asked whether the licences were paying for themselves, and a chart going down is not an answer that survives that question.

Leave it alone and push on review speed. Ask reviewers to turn work around within a day, add a second approver on critical paths, and run a review rota. This is the response everyone expects and it costs nothing politically.

The cost: review capacity is eleven people’s attention and no exhortation creates more of it. Faster review at a fixed capacity means shallower review, which is exactly the failure this kind of change is most vulnerable to, because a generated diff passes a shallow read trivially. The 19 per cent change failure rate would keep climbing and the cause would stay invisible, because no single incident would be traceable to the decision not to cap.

She also considered hiring two more senior engineers to review, and priced it: a five-month lead time at best, and the two most likely candidates would need three months in the codebase before their reviews meant anything. That is next year’s fix for this quarter’s problem.

There was one more complication. The fleet operator’s contract had six features named in it for the quarter. Capping work in progress would not reduce throughput in theory. In practice, nobody had ever run this team that way, and Meera could not promise the board that it would not.

How it played out

Meera capped it, at 15 rather than 12, and took the four numbers to the executive review instead of the activity chart. She framed it as a prediction that could be checked: same throughput, half the latency, in one quarter. By the end of the next quarter deployments per week had gone from 9 to 13, median lead time from 2.9 weeks to 1.1, and change failure rate from 19 to 12 per cent. Pull requests opened per week fell from 41 to 26, and the finance director asked about that number first. The contract’s six features shipped, five of them earlier than the quarter before. The thing Meera did not anticipate was that two engineers found the cap genuinely uncomfortable for the first month, because sitting with two open changes and nothing new to start felt like idleness, and she had to write down the rule for what you do when blocked: review, help finish something, or work on the constraint. Never start something else.

Worth arguing about

Commits and pull requests both rose sharply while deployments per week stayed flat. What does that pattern tell you about where the constraint sits, and why does adding more writing capacity make it worse?
One answer
It tells you the constraint is downstream of writing. Throughput is set by the slowest stage, and deployments per week is measured at the end of the pipeline, so a flat number with rising input means the extra work is accumulating in front of some stage rather than passing through it. Here that stage is review: eleven people with unchanged hours. Adding writing capacity raises the arrival rate at a queue whose service rate cannot move, so the queue grows without bound until the service quality drops to match, which is what a shallow review is. Delivery gets slower and less reliable while every activity number improves.
Meera could have hired two senior reviewers instead of capping. Using the Theory of Constraints sequence, why is that the wrong first move rather than simply a slower one?
One answer
The sequence is identify, exploit, subordinate, elevate, repeat, and hiring is elevation: the last step, not the first. Before adding capacity you make sure the constraint never idles and you subordinate the other stages to its pace. Capping work in progress is subordination: it moves effort to review automatically. Hiring skips two cheaper steps, takes five months plus three months of ramp-up, and adds reviewers whose reviews do not yet mean much in an unfamiliar codebase. It also leaves the arrival rate uncapped in the meantime, so the pile keeps growing while you wait.
Why did Meera present a prediction with a date rather than an argument about queueing theory, and what would have made her case unfalsifiable?
One answer
A prediction invites verification, which is what makes it persuasive to someone who is sceptical and who controls the budget: she named the four numbers, said which direction each would move and by when, and used data from the company’s own systems rather than an outside benchmark. An argument from principle cannot be checked, so it is heard as preference. Her case would have been unfalsifiable if she had promised better quality, higher morale or cleaner code (none of which has a number attached) or if she had left the definitions of incident and deployment free to be adjusted later, because then any outcome could be read as success.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
A team’s writing stage gets faster, so pull requests now arrive at 40 a week. The same eleven reviewers can service about 20 a week. What happens to the review queue?
2
In the 2025 trial, experienced maintainers were about 19 per cent slower with assistance and estimated afterwards that they had been about 20 per cent faster. Which mechanism best explains the direction of that error?
3
A randomised trial gives every participant the same task (implement an HTTP server in JavaScript) and the assisted group finishes about 55 per cent faster. What is the most precise claim that result supports?
4
Suppose cheap generation triples the amount of software an organisation runs over a decade. What does that imply about the amount of work available?
5
You are asked to generate a discount engine for a scheme nobody has written down. The code involved is short and simple. Why is this among the weakest cases for generation?
6
In a two-week self-experiment, why does the allocation have to be decided per task rather than per day or per week?

Reading what you did not write

10 lessons · 1 h 32

You will be able toReview a change whose author may not have read it closely: orient in an unfamiliar system fast, triage a diff by risk and cap it at a size where defect detection still works, judge tests and dependency changes on their own terms, and check the change against the stated intent rather than against its own internal consistency.

An illustrative case, built from documented patternsNineteen hundred lines, and a go-live on FridayA twelve-person company in Nashik that supplies a records system to four district hospitals

The system stores outpatient visits, prescriptions and lab results for four district hospitals. It has one reviewer with real depth in the prescribing module, Anil, who has been there five years, and a go-live date for a fifth hospital that was written into a contract with the state health department.

On Tuesday morning a colleague opened a pull request titled Add allergy warnings to prescribing. It changed 1,912 lines across 31 files. The description read, in full: “Implements allergy checking as discussed. Tests pass.”

The work was genuinely a day old. The author had spent Monday describing the feature, reading what came back, running it, and fixing the two places where it did not compile. He was not being careless by the standards anybody had ever set for him; the change worked in staging, the screen showed the warning, and the test suite was green with 94 per cent coverage on the touched files.

Anil opened it at nine and was still on the file list at twenty past. The list included the prescribing service, which he expected; a migration, which he expected; the allergy table, which he expected; and then a rewritten date formatter, a change to the session middleware, a new dependency for fuzzy string matching, and edits to eleven test files including four that had nothing to do with allergies.

He had about forty minutes before the morning standup and four other things to do that day. The go-live was on Friday and this feature was one of the three the health department had named.

The decision in front of him was which of two things to do, and both cost something.

Approve it, with comments. Read the prescribing service and the migration properly, skim the rest, and write a note saying so. The feature ships, Friday holds, and the department gets what the contract promised. This is what the schedule wanted and what the author expected.

The cost: he would be the first and only person to think hard about any of those lines, in forty minutes, across thirty-one files. He already knew from experience that past a few hundred lines he stopped comparing the before and after states and started reading for whether the code looked sensible, and it looked extremely sensible. The fuzzy matching dependency alone was a supply-chain decision he had not made, in a system holding prescription data.

Refuse it and ask for it in slices. A commit that adds the allergy table and backfills nothing. A commit that adds the lookup as a pure function with tests. A commit that wires it into prescribing. The date formatter, the middleware change and the dependency taken out entirely or argued for separately.

The cost: three or four review cycles instead of one, and on this team review turnaround was already two days because Anil was the only person who could do it. Four sequential reviews would be a week and a half, which is past Friday. Splitting is only free when review is fast, and here it was not, so refusing the diff would miss a contractual date, and the person who would have to explain that to the health department was not Anil.

He also noticed something in the deletions that he could not un-notice. The diff removed eleven lines, and one of them was a guard in the prescription save path:

if patient.allergies is None:
raise IncompleteRecord(patient.id)

The new code treated a missing allergy record as an empty list, so a patient whose allergy history had never been entered would produce no warning at all, silently, which is the same output as a patient with no allergies.

Nobody had asked for that change. It was the shortest path from a failing test to a passing one.

How it played out

Anil split the difference in a way that was specific rather than a compromise. He wrote in the review that he would approve the allergy table, the pure lookup function and the prescribing wiring, and that the date formatter, the middleware change, the fuzzy matching dependency and the four unrelated test files had to come out of this change and go somewhere else. That took the diff to about 600 lines, which he read properly the same day. He asked one question about the removed guard (not why it was removed, but what it had been protecting against and what protects against it now) and the answer was a shrug, which told him the change was not yet owned by anybody. The guard went back in, with a test asserting that a patient with no recorded allergy history blocks the prescription rather than passing silently. The feature shipped on Thursday. The fuzzy matching dependency was never re-proposed; the author had needed it for one call and wrote nine lines instead. Six weeks later the team added a rule to the contributing guide: any change that removes a validation, widens an exception, skips a test or drops a constraint says why in the description, and a script greps the deletions so the reviewer does not have to remember to look.

Worth arguing about

The change had 94 per cent coverage on the touched files and a green suite, and it still removed a guard that mattered. Why did neither the coverage number nor the suite object?
One answer
Coverage measures which lines executed, not whether anything would have noticed if they were wrong. The test that would have failed when the guard was removed was almost certainly the test that was changed or removed alongside it, and the remaining tests exercised the new path and asserted what it does. A suite derived from the same reading of the same request as the implementation encodes the same misunderstanding and confirms it. The only signals available were the deletion itself, which a human has to look for because attention follows additions, and the question about what the guard was protecting against, which had no answer.
Anil did not fully refuse the diff, even though the advice everyone knows is to insist on small changes. What made refusal the wrong move here, and what would have had to be true for it to be the right one?
One answer
Small diffs pay off when review turnaround is fast. On this team one reviewer meant a two-day turnaround, so four sequential reviews would run past a contractual go-live: splitting would have made delivery worse, not better. The conditional rule is that you fix review latency before you enforce diff size, because solving them in the wrong order makes both worse. Refusal would have been right if a second person could have reviewed the slices in parallel, or if the change had touched something irreversible (a migration dropping a column, or a payment path) where the cost of being wrong exceeds the cost of missing a date.
Seven of the thirty-one files had nothing to do with allergies. Why is that the first thing to check, before reading any code?
One answer
Because it is a fact about the change that takes four seconds to establish and it identifies the part nobody scoped. The ticket says what this change is for; the file list says what it touched. Anything in the second that is not explained by the first is work that was never requested, never sized and never thought about by a person, and it is where the unexamined risk sits: here, a middleware change in an authentication path and a new dependency in a system holding prescription data. Reading the code first spends the scarce attention before you know which files deserve it.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
Experienced reviewers have long relied on a fast check: does this look like something a competent person wrote? Why has that check stopped carrying information?
2
A diff adds a retry decorator with three attempts and exponential backoff to a function that posts a charge to a payment provider. Nothing in the code is locally wrong. What is the defect?
3
You have forty minutes and a 700-line diff. Which allocation makes the approval mean the most?
4
A test asserts fee(1000) == 1000 * 0.029 + 30, and the implementation is return amount * 0.029 + 30. What is wrong with this test?
5
Why are the deleted lines in a diff the highest-risk lines, and the ones human attention least reliably reaches?
6
A review bot leaves eight comments per pull request and two are useful. Why is this worse than having no bot at all?

Building oracles that stay ahead

10 lessons · 1 h 32

You will be able toAssemble a layered set of oracles for a change (properties and differential runs where behaviour must be preserved, types and database constraints that run for ever at no cost, mutation checks to prove the suite bites, and a staged release that can stop itself) and say for any change what would have to be true for it to be wrong and which layer would catch it.

An illustrative case, built from documented patternsThree weeks of shadow running, or a launch dateA payments company in Bengaluru replacing the settlement reconciliation engine that matches bank files against its own ledger

The reconciliation engine is nine years old, written by two people who have both left, and it decides which merchant payouts are correct. Every night it reads settlement files from four banks, matches them against the ledger, and produces a list of breaks for the operations team to work through in the morning. A wrong match is not a crash. It is a merchant paid twice, or not paid, discovered weeks later by somebody in finance.

The rewrite had gone unusually well. Two engineers produced a new engine in eleven days that was clearer, faster and about a third of the size. It had 420 tests and 96 per cent coverage, all green, and it agreed with the old engine on every case anybody had written down.

Priya, who was leading it, did one thing before proposing a date. She ran a mutation tool over the matching module, once, overnight.

The score came back at 41 per cent. Fifty-nine per cent of the mutants survived: change a comparison from less-than to less-than-or-equal, replace a return value with a constant, delete a statement, and the suite stayed green. The module with 96 per cent coverage was fully exercised and almost entirely unchecked. When she read the surviving mutants, the reason was obvious and slightly embarrassing: most assertions restated the implementation. assert fee(1000) == 1000 * 0.029 + 30 passes whether or not the formula is the one the bank uses.

That put a real decision in front of her, and the launch date was already in a slide deck.

Ship behind a flag in two weeks, with the suite as the oracle. Roll out to one bank first, watch the break count, move the rest across. The team is confident. The old engine stays in the repository for a month in case of a rollback.

The cost: the suite is not an oracle, and she now had a number proving it. A wrong match produces no error, no exception and no alert. It produces a payout that is slightly wrong, and the detection mechanism is a person in finance noticing an odd figure some weeks later. Change failure rate would not capture it; the deployment would look successful for a month.

Run the two engines side by side on real traffic for three weeks before switching. Take the settlement files for the last sixty days, run both engines over them, and diff every output. Then shadow the nightly run: both engines process the live file, the old one’s result is used, and disagreements are logged.

The cost: three weeks of two engineers doing nothing a roadmap recognises, plus about two hundred lines of throwaway scaffolding to run both and normalise the outputs. The launch slips past the quarter. The head of product had already told two large merchants that faster payouts were coming, and “we are verifying it” is a sentence that sounds like an excuse the second time you say it.

There was also a trap in the second option that Priya named before starting, because she had watched a colleague fall into it on a previous migration. The rewrite was supposed to fix two known bugs in the old engine. If they ran a differential comparison while also changing behaviour, every divergence would need a judgement call, and under time pressure the judgement becomes “accept”, which turns a fix into an indistinguishable regression.

How it played out

She took the three weeks and split the work so that the comparison could be exact. The rewrite went in as behaviour-preserving only; the two bug fixes were pulled out into separate changes to be made afterwards, each about twenty lines, each reviewed by a human. Sixty days of replayed files produced 1,340 divergences on about four million matched rows. Most were ordering: the old engine returned breaks in whatever order the database handed them back, so they normalised by sorting rather than by loosening the comparison. Eleven were real. Two of those mattered: a rounding difference on one bank’s three-decimal currency amounts, and a case where a settlement arriving twice in the same file was matched twice by the new engine and once by the old. The second one would have paid a merchant twice, at a rate of roughly one file in forty. Neither was in the 420 tests, and neither would have been, because nobody imagines a duplicate line in a bank file until they see one. The launch slipped by nineteen days. The scaffolding was deleted a month later. The team kept the replay corpus, and it is now the first thing any change to the engine runs against.

Worth arguing about

The suite had 96 per cent coverage and a 41 per cent mutation score. What specifically does that pair of numbers tell Priya that neither number tells her alone?
One answer
Coverage says every line ran during the tests. The mutation score says that for most of those lines, the code could be changed and no test would object. Together they identify the specific dangerous state: fully exercised and almost entirely unchecked, which looks identical to a strong suite on any dashboard. Coverage alone would have read as excellent. A low mutation score alone might mean the module is barely tested at all. The pair localises the problem to the assertions rather than to the reach of the tests, which is why reading the surviving mutants immediately showed the real cause: expected values computed by restating the implementation.
Why did Priya insist that the rewrite contain no behaviour changes, and what happens to a differential run when the two bug fixes are left in?
One answer
Differential testing verifies that the new thing matches the old thing. It is a perfect oracle only when the requirement is that behaviour must not change. With the bug fixes included, every divergence needs a human judgement about whether it is the intended fix or an accident, and there were 1,340 of them. Under launch pressure the judgement collapses into accepting differences one by one, at which point the fix and a regression are indistinguishable and the oracle has been talked out of its verdict. Splitting the work keeps the comparison exact, and leaves two twenty-line changes that a person can actually read.
Most of the 1,340 divergences were ordering differences. Why is normalising by sorting the right response, and what would have gone wrong with a comparison that ignored order?
One answer
Ordering differences here are genuine non-determinism in the old engine rather than a behaviour difference, so the honest move is to write one named normalisation (sort the output, canonicalise the keys) and keep the comparison exact everywhere else. Loosening the comparator to ignore order would widen it permanently, and every widening is a hole. An order-insensitive comparison would have hidden the duplicate-settlement bug, which shows up as the same break appearing twice: to a comparison that ignores ordering and multiplicity that looks like the same set of results, and the merchant gets paid twice.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
You ask for an implementation, then ask for tests for it. The suite is green. Why is that green run weak evidence?
2
What makes a property test expensive for pattern-matched code to satisfy by accident, in a way an example test is not?
3
A team runs a differential comparison between an old and a new pricing implementation during a change that is also meant to fix two bugs. What goes wrong?
4
A rule says a customer may have only one active subscription. Why does a partial unique index beat an application-level check?
5
Each test in a 500-test suite fails spuriously about one time in two hundred. What is the consequence for the build?
6
You want a canary to catch a fault that affects about one request in a thousand. The service handles two requests a second and the canary takes one per cent of traffic. What does a five-minute canary tell you?

Saying what you want

10 lessons · 1 h 27

You will be able toTurn a vague request into a written specification that serves as prompt, review criterion and test names at once: enumerate the ambiguities a competent stranger could not resolve, state the out-of-scope boundary, record the decisions that have no technically correct answer, and keep the whole thing in the repository so it cannot drift from the code.

An illustrative case, built from documented patternsAdd a waiting listThe admissions office of a self-financed engineering college in Coimbatore, three days before counselling opens

The request arrived as one line in an email from the registrar: Add a waiting list to the admissions portal.

Kavitha is the only developer on the portal. She has built things for this college for four years, she knows the code, and she could have a working waiting list by the end of the day: a table, a position number, an email when a seat opens. The screen would render, the registrar would see it working, and everyone would be pleased.

Instead she spent ninety minutes writing down the questions the sentence did not answer, which is a habit she picked up after a fee-refund feature two years earlier went out on an assumption and had to be unwound by hand across 300 students.

The list came to nineteen questions. Six of them turned out to be obvious. Four mattered enormously.

Ordering. When two students are waiting for the same seat, who gets it? The obvious implementation orders by the time the application was submitted. That is a decision with consequences nobody had discussed: applications are submitted online, so ordering by timestamp systematically favours students with better internet, and in this intake about a fifth of applicants applied from a browsing centre or a phone on a patchy connection. Ordering by merit rank is a different system, and ordering by merit rank within category is a third.

Category. Seats are allocated by reservation category, and the college’s own management quota has its own rules. Is there one waiting list or one per category? If a general-category seat opens and the next person waiting is in a reserved category, what happens? This is not a technical question. It is governed partly by regulation, partly by the college’s published prospectus, and the two do not use the same words.

Time. When a seat is offered, how long does the student have to accept before it passes to the next person? Twenty-four hours is defensible. So is forty-eight. The number determines whether the office can fill seats before counselling closes, and it has to be in the offer email, which means somebody has to decide it and own it.

Failure. The portal sends offer emails through a third-party service. If that service is down when a seat is released, does the seat sit unoffered until it recovers, or does the position advance and the student get told by phone? A silent advance while email is broken would move a student down the list for a reason that has nothing to do with them.

Kavitha had a decision of her own, and it was not about code.

Build it today on the obvious answers. Timestamp ordering, one list, twenty-four hours, and whatever the email library does on failure. It ships before counselling, the registrar is happy, and four decisions get made by default.

The cost: every one of those defaults is invisible when it is wrong. Nobody sees a tie-break rule. A student who loses a seat because their application arrived eleven minutes later on a slow connection never learns that is why, and neither does anybody else. If the category rule is wrong, it is wrong against a regulation, and the person who answers for that is the principal.

Get the answers first. That means the registrar, who is in meetings all week, and probably the admissions committee, who meet on Thursday. Realistically a day, possibly two.

The cost: counselling opens on Monday. A day lost is a day the office spends managing the waiting list on paper, which they have done before and hated, and Kavitha would be the reason. The registrar’s email did not read like an invitation to a requirements conversation.

How it played out

Kavitha sent the four questions in a numbered list, each with the options and one line on what each option would mean for a student, and she sent it as a reply to the registrar rather than as a ticket. She got answers in fifty minutes, because the questions were specific enough to answer between meetings and because the category question was one the registrar had been worrying about independently. Ordering is by merit rank within category, ties broken by application number ascending so that the order is stable between page loads. One list per category. Forty-eight hours to accept, stated in the offer email. If the email service fails, the seat is held and the office is alerted, because the college would rather a seat sit for two hours than a student lose their place to an outage. Kavitha put those four decisions in a file next to the code, with the reason for the fail-held behaviour written out, and turned each one into a test name. The build took a day and a half. In the first week of counselling the fail-held path fired twice. The line that made the difference later was the one recording why: in March a new contractor saw the branch that stops the queue when email is down, assumed it was a bug, and was about to make it advance.

Worth arguing about

Ordering by submission timestamp is the implementation nobody would have argued with. Explain why it is a specification failure rather than a coding failure.
One answer
The code would have been correct in the sense that it did what it was built to do. The failure is that a decision with real consequences for students was made by whatever was most convenient to implement, rather than by somebody with the standing to make it. Timestamp ordering encodes a rule (earlier submission wins) that systematically favours applicants with better connectivity, and the college had never chosen that rule. Ambiguity used to stop you while typing; here it would have been resolved silently and in a direction nobody examined, which is exactly the class of decision that has to be made before the code exists.
Kavitha wrote down a tie-break rule (application number ascending) even though ties are rare. What bug does that line prevent, and why is it hard to diagnose later?
One answer
Without a deterministic tie-break, two records with the same merit rank can come back in a different order on each query, because the database is free to return rows in any order it likes when nothing forces one. A student refreshing the page would see their position change, and paging through the list could show someone twice or skip them. It is hard to diagnose because it is intermittent, depends on the query plan and the data, cannot be reproduced on demand, and looks like a display glitch rather than an ordering fault. One line in the specification removes the whole class.
The failure question (what happens when the email service is down) had no technically correct answer. Why is that a sign the answer belongs in a written decision rather than in the code alone?
One answer
Because both options are defensible and the choice depends on what the institution can tolerate: holding the seat costs time in a short counselling window, advancing the queue costs a student their place for a reason unconnected to them. That is a business trade with a consequence, not an engineering fact, so the code alone records only which way it went and not why. The written reason is what stops a future change from reversing it, which is precisely what nearly happened in March, when a contractor read the hold branch as a missing else and was about to make the queue advance.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
For twenty years, the strongest argument against heavy specification was that writing the code was the thinking. What has weakened that argument?
2
Of all the questions in an ambiguity audit, which pair reliably catches the most, and why?
3
A criterion reads: a cart of ₹2,400 with code SAVE10 becomes ₹2,160. Why is that stronger than ‘applies a 10 per cent discount’?
4
A decision record is mostly valuable for one section that people routinely skip. Which, and what does it prevent?
5
What does adding ‘if the schema needs to change or a requirement is unclear, stop and say so’ actually change about what comes back?
6
You ask for a five-line behaviour change in a messy function and get back a clean rewrite of the whole function with the change inside it. The rewrite is genuinely better code. Why decline it as it stands?

The working loop

10 lessons · 1 h 29

You will be able toRun a change through a loop that keeps the cost of being wrong at fifteen minutes: steps small enough to throw away, git used so that reversal is instant, an agent confined so a mistake cannot reach production or your credentials, and an explicit rule for abandoning a conversation that has gone wrong twice.

An illustrative case, built from documented patternsThree hours on a SaturdayA two-person team building an order and stock tool for a knitwear exporter in Tiruppur, with a demo to the owner on Monday

Sanjay had one job for the weekend: make the packing list print the correct carton count when an order is split across two shipments. It is a small, annoying bug, it has been open for three weeks, and the owner of the business mentions it every time they meet.

He started at ten on Saturday morning with a clean tree and a plan that was three lines long. By one o’clock he had a diff of 900 lines across fourteen files and a feeling he recognised but had not yet named.

The path there was entirely reasonable at every step. The carton count came from a function that also computed the invoice weight, so that had to be untangled first. Untangling it revealed that the shipment model had no concept of a partial allocation, so he added one. The allocation needed a migration. The migration broke four tests, two of which were about something else entirely, so he adjusted them. One test about weight rounding kept failing, and the third attempt at a fix changed the assertion from 12.5 to 12.53, which made it pass.

He had not run the application since eleven.

At one o’clock he wrote down what was in front of him, honestly, which is the only part of this story that was deliberate.

The diff had grown past the point where he could say what was in it.
Nothing had been executed for two hours.
A test assertion had moved to match the code, and he could not now say which of the two was right.
Two suppression comments had appeared to quiet type errors around the new allocation model.
A new dependency for decimal handling was in the manifest and he did not remember agreeing to it.
He could not describe the change in one sentence without the word “and”.

The demo was on Monday morning. The owner had been told the carton count would be fixed. The decision was which of two things to do with the rest of the weekend, and both cost something he cared about.

Push on. Another three or four hours would probably get it green. The refactor underneath was arguably an improvement: the shipment model genuinely did need partial allocations, and he would have had to do it eventually. Monday holds, the owner is satisfied, and the work is real.

The cost: he would be shipping fourteen files of change to an order system he maintains alone, containing a weight calculation he no longer trusts and a test he edited to agree with it. If the rounding was wrong, the error appears on a commercial invoice going to a buyer in Europe, and the person who finds out is the buyer. His partner, who reviews his changes on Mondays, would be reading 900 lines in an hour, and they both knew what that review would actually be.

Go back to the last commit that worked. That is git checkout . and three hours gone. Then do the original three-line plan against the existing model, with no untangling, no migration, and no allocation concept, which means the packing list code stays ugly, because it was ugly for reasons.

The cost: three hours of a Saturday, and the acknowledgement that the afternoon produced nothing. There was also a genuine loss in it. Some of what he had learned about the shipment model was real and would be useful, and throwing the code away felt like throwing that away too.

How it played out

He went back. The reset took four seconds and the fix took fifty minutes: two lines in the carton count and one test that he watched fail before he made it pass. It shipped on Sunday evening and the demo went fine. He kept one thing from the abandoned afternoon: a note in a scratch file saying that the shipment model needs partial allocations, that this is a real piece of work, and that it is not a Saturday job attached to a bug fix. That became a separate change three weeks later, done on its own, behaviour-preserving, with the rounding question settled first by asking the owner what the buyer’s invoice actually has to say. The rounding turned out to be neither 12.5 nor 12.53: the buyer’s contract specifies weights to two decimal places rounded half up, and the existing code was rounding half to even, which had been quietly producing a one-paisa-scale discrepancy on about one invoice in nine for two years. He found that by asking a person, not by running the tests. The two-line note at the end of every long session is now a habit, and after two months the same pattern had appeared four times: every one of his bad sessions began with a fix that required a refactor first.

Worth arguing about

Sanjay’s afternoon contained six warning signs. Which one is the earliest reliable indicator, and why is that the one to act on rather than the later ones?
One answer
The diff growing while he stopped reading it. It is the earliest because it appears before anything breaks, and it is reliable because it is a fact he can check in four seconds rather than a judgement about quality. By the time the later signals arrive (tests moving to match code, suppressions appearing, being unable to describe the change in a sentence) the cost of going back is hours rather than minutes. Acting at the first signal costs ten minutes: stop, run something, read the diff, commit what works and discard the rest.
He felt he was discarding value by resetting. Explain why the three hours were not an asset, and what genuinely was worth carrying forward.
One answer
The code either works and is understood or it does not, and this code was neither: he could not say whether the weight calculation was right, and he had edited a test to agree with it. A nearly-working change that has resisted correction is evidence about the framing rather than a measure of distance remaining, and the feeling of closeness is exactly what makes the trap work. What was worth carrying forward was the knowledge (that the shipment model needs partial allocations and that this is a separate piece of work) which is why he wrote it down in one line and threw the rest away.
The rounding question was settled by asking the business owner, not by reading the code or running the suite. What does that say about where the oracle for this behaviour lives?
One answer
The correct value came from outside the system entirely: the buyer’s contract specifies two decimal places rounded half up. No test could have established it, because both the implementation and any test derived from it encode the same assumption, and the suite had been green for two years while the code rounded half to even. This is the class of rule that exists in a document and in a person’s head rather than in any corpus or codebase, and until somebody goes and gets it, every assertion about the weight is just the implementation repeating itself.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
Why commit or stash before letting a tool edit files unattended?
2
What does git bisect run require of you before it can find the commit that introduced a failure?
3
An agent runs in a container with the project mounted, and the container’s environment still carries a deployment token. What has the sandbox bought you?
4
It has been wrong twice in the same way. Why is a third correction usually more expensive than starting over?
5
You ship a working service in a language you do not know. Which category of mistake is most likely to survive both the compiler and the test suite?
6
Someone with no budget asks what to set up first for a free workflow. What is the correct ordering and why?

Teams, process and delivery

10 lessons · 1 h 30

You will be able toChange a team’s process to match the new constraint: measure delivery with the four keys rather than activity counts, cap work in progress at the bottleneck, put verification in the plan with a named owner and real hours, schedule consolidation, defend coherence with automated fitness functions, and write a one-page tool policy that survives a deadline.

An illustrative case, built from documented patternsThe day nobody could bill forA 40-person services firm in Hyderabad maintaining a claims system for an insurance client, billed by the feature

The contract is the important fact. The firm is paid per delivered feature against a quarterly statement of work, and time spent on anything not named in that statement is time nobody pays for. This arrangement has been in place for four years and it works well enough that neither side has wanted to change it.

In the past eighteen months the team’s output on paper has roughly doubled. So has something else. Ravi, the engineering manager, ran two commands on the repository before a quarterly planning meeting, mostly out of curiosity.

The first counted function names that appear more than once. The second was a copy-paste detector. Between them, on a codebase of about 240,000 lines, they found: five distinct date formatters, three HTTP retry helpers with different backoff behaviour and one of which retries POSTs, four separate implementations of the policy-number validation rule, and two ways of representing a claimant, one of which was introduced eleven months ago and now has 60 call sites.

None of that was anybody’s fault in a way you could point at. Generating a new 200-line helper is cheaper than finding the existing one, reading it and deciding whether it fits. Every one of those additions passed review, because a reviewer looking at a correct new function has no specific reason to object, and “this is the third way we do this” is the review comment people swallow because it feels pedantic.

The cost showed up in incident records rather than in anything aesthetic. In the previous quarter a date parsing bug on the claim-received field had been fixed twice and missed twice, because the person fixing it found two of the five formatters. The second miss caused a batch of 900 claims to be assigned the wrong assessment deadline, which the client noticed.

Ravi wanted one day a month, every engineer, deletion only. No new features. Merging implementations, removing dead code, deleting stale flags, retiring the abandoned reporting module that still has a nightly backup.

The decision was his to propose and the client’s to accept, and both answers cost something.

Take the day. Twelve days a year across 40 people is about 480 person-days, which at the contracted rate is a visible number on a spreadsheet. The client is not paying for it, so either the firm absorbs it or the statement of work gets fewer features in it. The account manager’s view, stated plainly in the meeting, was that a services firm that delivers less this quarter to be faster next year is a services firm that loses the renewal to one that does not.

Leave it and keep shipping. Nothing breaks immediately. Duplication does not cause an outage; it raises the cost and the failure probability of every future change in that area, permanently and quietly. Ravi could not name the date on which it would become unaffordable, which is exactly why it had never been scheduled.

He also knew which argument would not work. He had tried “the codebase is getting messy” the previous year and watched it lose to a feature, as it always does.

How it played out

He argued it with incident arithmetic instead of with elegance, and he brought two numbers the client already trusted. Five date formatters means a date bug has to be fixed five times, and last quarter it was fixed twice and missed twice, and the second miss was the 900 misassigned deadlines the client had raised. The abandoned reporting module is 6,000 lines, two dependencies with open advisories, and one table backed up nightly that nobody reads. The client agreed to one day a month for two quarters, on the condition that it was measured and reported. The first day removed 11,000 lines, two dependencies and nineteen stale feature flags, and merged the five date formatters into one, which took most of the day and turned up a sixth nobody had found. Ravi published the numbers, because that day is the one day of the year on which lines removed is a legitimate metric. What changed the argument for the third quarter was not the line count: it was that the change failure rate on claims processing went from 16 to 9 per cent, and the client’s own operations team noticed before the report was written. The habit that stuck cost nothing at all: before adding a helper, search for one, and if you write a new one anyway, put one sentence in the pull request saying why the existing one did not fit.

Worth arguing about

Every one of those duplicate helpers passed review. What is the structural reason, and what single norm makes the duplication decision visible without slowing review down?
One answer
A reviewer looking at a new, correct function has nothing specific to object to: adding is locally justified and the cost lands somewhere else and later. Deleting or reusing requires knowing what already exists, which requires understanding the whole system, so the default direction is accumulation. The norm that fixes it is one line in the contributing guide: before adding a helper, search for one, and if you write a new one anyway, say in one sentence why the existing one did not fit. That costs the author thirty seconds, gives the reviewer something concrete to check, and is the only point in the process where the decision is visible to anybody.
Ravi’s argument the previous year (that the codebase was getting messy) lost to a feature. Why did the incident version win, and what does that suggest about how to make this case generally?
One answer
Messiness is an aesthetic claim with no number attached and no cost the client recognises, so it loses to anything with a delivery date. The incident version converted duplication into arithmetic the client already believed: five formatters means a bug fixed five times, last quarter it was fixed twice and missed twice, and one of those misses was the incident they had already complained about. It also had a security component (6,000 dead lines and two dependencies with open advisories) which is a category that gets budget. The general lesson is to argue consolidation in the currency the other side already tracks: incidents, failure rate, attack surface, and the cost of the next change.
The firm measured lines removed on deletion day and treats that as legitimate, while the course warns against lines of code as a metric everywhere else. Reconcile those two positions.
One answer
Lines of code fails as a productivity metric because it rises with tool use regardless of what is delivered, and because measuring it pushes people to produce more, larger, less carefully read changes into a queue that is already the constraint. On a deletion day the incentive points the other way: the only way to score is to remove code, which is the behaviour you want and which nothing else in the process rewards. It is also bounded to one day and reported as an event rather than used to assess individuals. The metric is not good or bad in itself; what matters is the behaviour it produces in the context where it is applied.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
What property makes the four delivery measures hard to game, which commits and pull requests lack?
2
Commits and pull requests are up sharply, deployment frequency is flat, lead time is up and change failure rate is up. What does that signature mean?
3
A team merges about 20 changes a week and has 25 open at any moment. What is the time from opening to merged, and what happens if they cap in-flight work at 12?
4
A demo works on day one and the feature then appears to stall for three weeks. What is the useful planning response?
5
You want a day a month for consolidation and the roadmap has features on it. Which argument actually works?
6
Why does a policy that says ‘every line of every change must be understood by the reviewer’ fail, while a risk-tiered one survives?

Skill, hiring and the missing rung

9 lessons · 1 h 24

You will be able toDiagnose where skill formation has been short-circuited and design around it: separate fluency from understanding with a test you can run in five minutes, name the parts of the craft whose feedback loop a generator cannot close, construct an apprenticeship task that is still expensive to generate, and run a hiring loop that discriminates between candidates who all have the same tools open.

An illustrative case, built from documented patternsA slow endpoint and a week of somebody’s timeA nine-person engineering team at an exam-preparation company in Jaipur, five weeks before the traffic peak

The results page had been getting slower for two months. At the start of term it responded in about 300 milliseconds; it now takes between four and eleven seconds for students with more than about forty attempted tests, and the support inbox reflects it. In five weeks the mock-exam season starts and traffic goes up by a factor of six.

Divya, who leads the team, could see two ways to spend the next week.

Give it to Arjun, the senior engineer. He has been on the system three years. He would open the query plan, see the problem in about twenty minutes, and have a fix by the afternoon. The cost is four hours of a person whose week is otherwise full, and it is off the board before the weekend.

Give it to Nikhil, who joined four months ago out of college. Frame it as a diagnosis rather than a fix: find out why it is slow, come back with evidence and a recommendation. Pair him with Arjun for the first hour and then leave him to it. Realistically a week, perhaps more.

The second option costs a week of a junior’s time on a problem that could be closed in an afternoon, plus a few hours of Arjun’s attention in pieces, five weeks before the busiest period of the year. If it runs long, they are fixing a performance problem under load instead of before it.

It also costs something less obvious, which Divya had thought about more than she expected to. Nikhil was doing well by every visible measure. He had shipped more in four months than she had in her first year, his changes were clean, his pull requests were small and his tests passed. And in a review three weeks earlier she had asked him why a particular retry wrapper was on a function, and he had paraphrased the code back to her. Not wrongly: he described exactly what it did. He could not say what would happen if it were not there, or why that endpoint was not idempotent.

That is not a criticism of Nikhil. The work that used to make people senior (the small bug that costs three hours and teaches you how the scheduler works) is the work that got cheapest, and his four months had been full of tasks that were real but cheap to get wrong. He had shipped a great deal and had not yet been made to find anything out.

Divya checked her own assumption before deciding, because she was aware she might be romanticising her own first year. She looked at the last twenty changes Nikhil had merged. Every one of them was a change to code whose behaviour was already decided: a field added to a form, a filter on a list, a copy change, a test backfill. Not one of them had required him to work out what should happen. That was an allocation she had made, one ticket at a time, without ever deciding it.

There was a third consideration pulling the other way. The company had two other juniors, and the market was such that Divya was not confident of replacing any of them quickly. If Nikhil spent two more years shipping well and learning slowly, he would look fine on every metric right up to the first week he was asked to do something the tools could not do, probably during an incident, probably at night.

How it played out

She gave it to Nikhil, with a deadline of Friday and a rule that he was to come back with a diagnosis rather than a patch. Arjun sat with him for the first hour and did one thing: showed him how to get the query plan out of the database and how to read the difference between an estimate and an actual row count. Then he left. Nikhil spent Tuesday convinced it was the front end, because that is where the slowness was visible, and lost most of a day to it. On Wednesday he found that the results page issued one query per attempted test rather than one query in total, which had been true for a year and had not mattered when students had five attempts. On Thursday he found the part that Arjun would probably have missed, because Arjun would have stopped at the first answer: a second endpoint, used by the weekly progress email, had the same pattern and ran against every active student every Sunday night, and it had been quietly taking ninety minutes and finishing before anybody was awake. Under six times the traffic it would not have finished at all. The fix was two queries and an index. It took him three hours on Friday, and he wrote up the diagnosis: the symptom, the false start, the query plan, the two endpoints, and one sentence saying what would have caught it earlier, which was that nobody looks at the plan for a query until it hurts. The team added a check that fails the build on a query issued inside a loop over a result set. Divya’s view afterwards was that the week cost about four extra days of delivery and bought a person who now reaches for the instrument instead of guessing, and that she would not have been able to justify it in a quarter with a harder deadline.

Worth arguing about

Nikhil looked strong on every visible measure and could not explain a retry wrapper he had shipped. Why is shipped output the metric most likely to mislead here?
One answer
It is the measure the tools inflate most directly. Producing a correct, clean, passing change no longer requires having constructed an understanding of why it is correct, so volume and quality of output stop being evidence of the learning that used to come with them. Reading generated code requires recognition rather than recall, and recognition feels the same from the inside, so neither the manager’s dashboard nor the engineer’s own confidence registers the gap. The gap becomes visible only when somebody is asked to explain a line or to do something the loop cannot do, which is why asking for the explanation is the instrument, and it is a conversation rather than a metric.
Divya framed the task as a diagnosis rather than a fix. What property makes that a good apprenticeship task in a way that ‘fix the slow endpoint’ is not?
One answer
The expensive part is figuring out what to do rather than doing it. Generation collapses the writing (the eventual fix was two queries and an index) but it does not collapse reproducing a symptom, reading a query plan, eliminating a false hypothesis, or noticing a second code path with the same shape. Handing over the fix would have handed over the cheap part and kept the part that teaches. The diagnosis framing also makes the deliverable a piece of evidence and a write-up, which is checkable and presentable, so the learning is visible to somebody who can correct it.
Nikhil found a second endpoint with the same pattern that a faster investigator would probably have missed. What does that say about the relationship between speed and the value of the exercise?
One answer
Arjun would have gone to the query plan first, found the per-attempt query, fixed it and stopped, because an experienced diagnosis is efficient and efficiency here means stopping at the first sufficient cause. Nikhil’s slower, broader search (including the day lost to the front end) covered ground that a targeted search skips, and that is where the Sunday email job was. The point is not that slow is better; it is that the exercise bought two things at once, a person who can now read an instrument and a finding that would not otherwise have surfaced until the job failed to complete under six times the load.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
What is the mechanism by which reading fluent generated code produces less learning than writing the same code yourself?
2
In the Roediger and Karpicke study, the students were also asked to predict their own performance. Why does that detail matter more than the retention numbers?
3
In the illusion of explanatory depth, at what point do people’s confidence ratings drop, and what does that imply for checking your own understanding of a change?
4
What is the selection rule for deciding where to spend study time now?
5
What property should a task have to be worth giving to a junior engineer now?
6
In an interview run with the tools switched on, which candidate behaviour is the clearest negative signal?

Provenance, security and the unsettled law

10 lessons · 1 h 32

You will be able toSecure a change whose author may have been a model: verify a dependency before installation rather than after, name the vulnerability classes that arrive by default and the one class no scanner can find, break the lethal trifecta so an injected instruction cannot both reach secrets and send them anywhere, and state the shape of the copyright disagreement across jurisdictions without pretending it is settled.

An illustrative case, built from documented patternsThe export that ran laterA two-person company in Nagpur selling a reporting tool to about 60 diagnostic laboratories, with a contracted delivery on Monday

Each laboratory is a separate tenant. They upload test results, and the tool produces the monthly summaries that laboratory owners send to referring doctors. The data is patient test results with names and phone numbers attached.

The new feature was a CSV export: pick a date range and a report type, get a file. It had been asked for by three laboratories and promised to one in writing, with a date, because that laboratory was deciding between this tool and a competitor.

The implementation arrived in two parts over an afternoon, and each part was correct on its own. A request handler on the reports page that validates the date range, checks the session and queues an export job. And a worker that picks the job off the queue, runs the query, writes the file and emails a link.

On Thursday evening Farah ran the four questions she asks of anything that touches laboratory data before merging it. It takes about two minutes.

Where does untrusted input enter? The date range, the report type, and a filename the user can set.

What can this code do that its caller cannot? The worker runs as a service account with read access to every tenant’s results, because it is one worker serving all 60 laboratories.

Who is allowed to do this, and which line checks it? The handler checks the session and the tenant at line 38. The worker, which runs some seconds later from a queue, does not check anything. It has a laboratory identifier in the job payload and it trusts it.

If this is wrong, what does an attacker get? A logged-in user at any laboratory who changes one number in the queued payload gets another laboratory’s patient results, with names and phone numbers.

The finding took ninety seconds. The scanner in the pipeline had not objected, and would not have: there is no pattern to match, because the query is a perfectly ordinary query and the rule it violates (a laboratory may see its own results and no others) exists in a contract and in Farah’s head, not in any code or corpus.

The decision was about Monday.

Fix it properly before shipping. The worker re-reads the tenant from an authoritative source and re-checks it, the job payload stops being trusted, and there is a test where laboratory B requests laboratory A’s export and receives a refusal. Farah estimated a day and a half, because the queue payload shape is used by two other jobs that would have to change with it.

The cost: the delivery date is Monday and a day and a half from Friday morning lands on Tuesday at best. The laboratory choosing between two products has a demo booked. Her partner’s view was that a two-person company that misses its first contracted date with a new customer does not usually get a second one.

Ship it and fix it next week. The gap requires a logged-in user to tamper with a queued payload. Nobody had done it, nobody knew the feature existed yet, and the laboratories are small businesses rather than adversaries.

The cost: patient test results under a health data obligation, sixty tenants, and a defect whose exploitation leaves no trace distinguishable from ordinary use. If it were used, the first person to find out would probably be a laboratory owner receiving a competitor’s patient list.

How it played out

They shipped on Monday with the feature restricted, which was a third option neither of them saw until they stopped arguing about the first two. The export went live for a single tenant (the laboratory that had been promised it) with the worker hard-coded to that one identifier and a check that refuses anything else. That took forty minutes. The demo happened on the promised date on real data. The general version, with the worker re-reading the tenant from the database and refusing a mismatch, shipped nine days later with the negative test attached. Two things came out of it that outlasted the feature. The four questions became a required line in the pull request template, because the finding had cost ninety seconds and the fix would have cost a week if it had been found in production. And they added the pattern to the conventions file in the repository (every background job re-establishes the tenant from an authoritative source and never trusts its payload) after Farah checked the two other jobs using that queue and found that one of them, written eight months earlier, had the same gap and had been live the whole time.

Worth arguing about

The handler checked the tenant and the worker did not, and each half was correct in isolation. Why is this shape particularly common now, and why does no scanner find it?
One answer
The two halves are usually produced separately, each from a request that makes sense on its own: a handler that validates and queues, and a worker that processes a job. Neither prompt contains the fact that authorisation established at request time does not survive into a job executed later by a service account with broader rights. A scanner cannot find it because there is no pattern to match: the query is ordinary and the code is valid. The rule it violates, that a laboratory sees only its own results, exists in a contract and in somebody’s head rather than in the code, so there is nothing for a static analyser to compare against.
Farah’s third option shipped the feature to one tenant on the promised date. What made that a genuine resolution rather than a fudge?
One answer
It removed the vulnerable capability rather than accepting it: with the worker refusing any identifier but one, the tampering path does not exist, so the risk is not deferred but eliminated for the period the restriction is in place. It also cost forty minutes rather than a day and a half, because it did not require changing the queue payload shape used by two other jobs. The general version then shipped nine days later with the negative test, on its own, reviewed properly. It is the same move as separating a behaviour-preserving change from a behaviour change: reduce the scope until the part you must get right is small enough to get right.
Reviewing the two other jobs on that queue found an eight-month-old instance of the same gap. What should a team take from that, beyond fixing the one job?
One answer
That a defect found once in a shape is worth searching for across every instance of that shape, because the mechanism that produced it was general rather than a one-off lapse. The durable output is not the fix but the written convention (every background job re-establishes the tenant from an authoritative source and never trusts its payload) placed in the conventions file where both the team and the tools will read it, plus the negative test that would have caught it. An incident or a near miss is worth one permanent check in the same week; otherwise the knowledge lives only in the person who happened to look.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
An invented package name is a broken build. What turns it into a security problem?
2
An endpoint returns an invoice by identifier and checks that the caller is logged in. Why will no scanner report the missing ownership check?
3
An agent has repository access, reads issues filed by the public, and can open pull requests. Which single change most reduces the risk?
4
Why configure a static analyser to block only on findings the change introduced?
5
Why is watermarking generated code not a workable provenance control?
6
The lawfulness of training on copyrighted material is unresolved and differs by jurisdiction. What planning rule follows?

What did not change, and what to do now

9 lessons · 1 h 21

You will be able toSeparate what got cheaper from what did not: identify which decisions in a design are one-way doors worth slowing down for, estimate a change by its ten-year cost rather than its writing cost, state what it would cost to leave a tool before adopting it, and name which questions in this field are genuinely open along with the evidence that would settle each one.

An illustrative case, built from documented patternsThe identifier nobody choseA company in Ahmedabad running fee collection for about 700 schools, three years after the first line of code

The product works. About 700 schools, a little over three million payment rows, and a business that grew faster than anybody planned for. It is also slow in a way that has been getting steadily worse, and in February the team finally measured why rather than guessing.

Inserts into the payments table had gone from about 4 milliseconds to about 40. The table’s primary key is a random identifier, generated in application code, and every insert lands in a random position in the index rather than at the end. On a small table that costs nothing. On three million rows with the index no longer fitting comfortably in memory, every insert is a page read, a page split and a write, and the fragmentation compounds. Fee collection is bursty (a school opens its payment window and thirty thousand parents pay in two days), so the slow path is the path that matters most.

Nobody had chosen the identifier scheme. In the first week of the project, three years earlier, a generated migration had used a random identifier because that is the common default, it was correct, it passed review, and there was no reason to argue with it. Two weeks later there were ninety files depending on it. There is now a decision nobody made sitting underneath everything.

The options were priced properly, which took a fortnight of somebody’s time and was the most useful part of the exercise.

Migrate to a time-ordered identifier. Inserts land at the end of the index again. The estimate was six weeks of two engineers, because the identifier appears in eleven foreign keys, in the payment gateway’s reconciliation files, in receipt PDFs already issued to parents, and in the export three schools have built their own reporting against. Some of those identifiers are printed on paper in people’s homes. The migration would need a period where both schemes exist, and a plan for what a receipt from last year means.

The cost: six weeks in which no school gets anything new, in a market with two competitors actively selling. And a migration on a live payments table carries its own risk, which is not zero and is hard to describe to a board.

Do not migrate. Reduce the pain instead: partition the table by academic year, rebuild the index periodically, add memory to the database. Each of these buys time and none of them removes the cause.

The cost: the arithmetic gets worse with every school added, which is the thing the company is trying to do. The measures are recurring work and recurring spend, and at some point during a busy fee window the database does not keep up, which is the worst possible moment to discover the limit.

There was a third thing, which is the one worth taking from this case. While pricing the migration, the team wrote down every other decision of the same kind that had been made implicitly in the first month: how time is stored, whether a parent can belong to two schools, what happens to a receipt when a school leaves, and what personal data the payment records hold. Two of those turned out to be fine. One (timestamps stored without a zone, on a system now serving two schools in the Gulf) was quietly wrong and had been for eight months.

How it played out

They did not migrate. They partitioned by academic year, added memory, and scheduled an index rebuild after each fee window, which brought inserts back to about 9 milliseconds and bought an estimated two to three years at the current growth rate. The decision was written down as a decision rather than left as a state of affairs: a one-page record naming the cost of migrating, the cost of not migrating, the measures taken, and the specific number (sustained insert latency above 25 milliseconds during a fee window) at which the question gets reopened with the migration funded. What changed permanently was the front of the process. New tables get a time-ordered primary key, and the first month of any new piece of work now includes an explicit half hour on the five decisions that do not swing back: the identifier scheme, how time is represented, the tenancy model, the shape of anything a third party will call, and what data is collected. The timezone problem was fixed in three weeks, before the Gulf schools grew from two to nine, and it was found only because somebody sat down and listed the doors that had already been walked through.

Worth arguing about

Nobody decided to use a random primary key, and yet it became the most expensive property of the system. What is the mechanism, and where in the process should the check have been?
One answer
An irreversible decision was made implicitly because something had to be picked, and what gets picked is whatever is most common in public code. Two weeks later ninety files depended on it and it was load-bearing. The cost of a wrong one-way door has not changed; what changed is how fast you pass through one without noticing. The check belongs at the start of a piece of work rather than at review, because by review the decision is already distributed across the diff: before designing, ask what reversing each choice costs in eighteen months with millions of rows and external integrations, and write down the few whose answer is weeks.
The team chose not to migrate, which is the option that leaves the known defect in place. What makes that a defensible decision rather than an avoidance of one?
One answer
It was priced on both sides rather than deferred: six weeks of two engineers against a measured slowdown, with the cost of migrating inflated by identifiers printed on receipts in people’s homes and embedded in three schools’ own reporting. The mitigations bought two to three years and were costed as recurring work. Crucially they wrote down the decision, the rejected alternative and the specific threshold at which it gets reopened, so it is a choice with a trigger rather than a thing that was never faced. A decision is avoided when nobody names the cost of the path not taken and nothing brings the question back.
Listing the other implicit decisions turned up a timezone fault that had been live for eight months. Why is that category of bug so likely to stay hidden, and what made the audit find it?
One answer
Storing a timestamp without a zone produces no error, no exception and no failing test. It produces a value that is right for most users and wrong by a fixed offset for the ones elsewhere, so it is invisible until somebody compares two records or a customer complains, and with two schools in the Gulf out of 700, nobody was comparing. The audit found it because it asked a different question: not what is broken, but which decisions were made implicitly and cannot easily be reversed. That question reaches faults defined by their silence, which is exactly where study time and deliberate attention pay off.

Test yourself6 questions on this modulePractice. Nothing is recorded and no score is kept.

1
Two teams adopt the same tools. One reports large gains and the other none at all. What does the essence and accident distinction predict about the difference?
2
What changed about one-way doors, given that reversing them costs exactly what it always did?
3
Which property most reduces what a system costs to keep, as distinct from what it costs to write?
4
Where does lock-in to a tool actually accumulate?
5
Bank teller employment rose for decades after cash machines arrived, while telephone operators disappeared. Which pair of variables separates those outcomes?
6
The course marks several headline questions as genuinely open. Which of these is not open, and is established enough to build practice on?

How Software Development Changed

9 modules. 87 lessons.

Where the cost actually went

Reading what you did not write

Building oracles that stay ahead

Saying what you want

The working loop

Teams, process and delivery

Skill, hiring and the missing rung

Provenance, security and the unsettled law

What did not change, and what to do now

What this course is

Who wrote it

More in Use it in your work