Shipping It

Sharma Classes runs a weekly test series for about four thousand students. The papers are taken on phones, in a two-hour window on Sunday morning, on a site the centre’s one part-time developer built and keeps running on a rented virtual machine.

The old machine was small and had fallen over twice during a paper. So on Saturday evening the developer built a bigger one, copied the application across, restored the database, checked the site by typing the new IP address into his browser, and at half past ten at night changed the A record for tests.sharmaclasses.in from the old address to the new one.

At six on Sunday morning the phone started ringing.

Some students were seeing the new site, with Sunday’s paper on it. Others were seeing the old site, which was still running, still showed last week’s paper, and had a database nothing was writing to any more. A few were getting a full-page browser security warning. Nobody could work out what separated the three groups, because the same student got a different answer on the hostel wifi and on mobile data.

Three facts were established in ten minutes, with one command each:

• dig +short tests.sharmaclasses.in @ns1.provider.net: the authoritative nameserver returned the new address. The change had saved.
• dig +short tests.sharmaclasses.in @1.1.1.1: a public resolver returned the old address. It had asked before the change and was still holding the answer.
• dig tests.sharmaclasses.in: the record’s TTL was 86400. One day. Nobody had touched it, because it was the default the provider set.

Nothing propagates. Records are pulled and then cached, and every cache holds its copy until the TTL runs out. A resolver that asked at nine on Saturday evening would keep handing out the old address until nine on Sunday evening: the whole of the test window and most of the evening after it. Lowering the TTL at six in the morning changes nothing for a cache that is already holding a copy, because the copy it holds carries the old number with it.

The security warning turned out to be a different fault wearing a DNS costume. The certificate on the new machine had been issued for tests.sharmaclasses.in and nothing else. Students who typed www.tests.sharmaclasses.in, which the old machine had also answered for two years, were meeting a name the new certificate did not cover.

So at ten past six, with the paper opening at nine, there were two options and a real cost on each.

Turn the old machine off. Then every student still resolving to the old address gets nothing at all: a connection refused, for some of them until that evening. It is clean, it is honest, and it would have stopped a large share of four thousand students from sitting a paper they had paid for.

Leave the old machine running and make it forward. Put a reverse proxy on the old box that passes every request through to the new server, so both addresses serve the same site and the same database. The costs are not zero: an extra network hop for everyone still on the old address, a second machine to pay for and to remember to turn off, and a certificate on the old box that has to be valid for every name it answers to, including the www name that was already broken.

The centre’s owner asked a third question, which was whether to postpone the paper by a week. Four thousand students had planned a specific Sunday around it, several had travelled, and a postponement is not free either.

The clinic books about ninety appointments a day. Reception starts taking them at half past seven in the morning; the doctors see patients from eight. The system is a small web application on a container platform, running two instances, maintained by the receptionist’s nephew Rohit in the evenings.

Dr Kulkarni had asked for something reasonable. The patients table has a column called name, which is printed on the prescription and shown at the top of the booking screen. She wanted the screen to show what the patient is actually called (Baby, Amma, a short form) while the prescription keeps the legal name. Rohit’s plan was to rename name to legal_name, add display_name, and update the code.

He had Tuesday from eight until ten. The change was one migration and about forty lines.

What stopped him was reading the platform’s deploy documentation properly for the first time. It performs a rolling deploy: it starts a new instance, waits for its health check, sends traffic to it, then stops an old one. For somewhere between thirty and ninety seconds, both versions of the code are running and both are serving real requests.

So shipping the rename and the code together does this. The migration runs. Every request that lands on an instance still running the old code asks for a column called name, which no longer exists, and fails. Reception is not open at nine at night, but two of the doctors’ own phones poll the day sheet, and the pharmacy next door uses the same login.

He found a second problem while looking. The container’s start command was npm run migrate && npm start. With two instances that is two migrations running at the same moment against the same database. It had never caused trouble because every previous migration had been an added table. A rename applied twice does not fail politely.

And a third. The clinic runs a monthly report query that reads the whole patients table and takes about four minutes. An ALTER TABLE wants an ACCESS EXCLUSIVE lock, which conflicts with a plain SELECT. If the report is running when the migration asks, the migration waits, and every query on patients that arrives afterwards queues behind the waiting lock request. A change that takes one millisecond takes the clinic’s booking system offline for as long as the report runs.

That left a decision with a cost on each side.

One deploy tonight. It is done, the doctor gets the feature she asked for two weeks ago, and it takes ninety minutes. The price is a window of failures during the rollout, no safe rollback, because rolling the code back leaves a schema the old code cannot read, and a migration that could stall behind the report.

Expand, migrate, contract. Add display_name as a nullable column tonight and write to both. Backfill in batches. Switch reads on Thursday. Stop writing the old column the following week. Drop it a fortnight later. Nothing ever breaks and every step is individually reversible. The price is three evenings instead of one, a feature the doctor has already waited for, and a conversation Rohit did not want to have, because the previous change had also been late and she had said so.

The portal takes scholarship applications from students at about two hundred government colleges. It runs on four instances behind a load balancer, in front of a managed Postgres. The deadline was the coming Saturday, which is when almost all the traffic arrives.

On Friday morning a clerk in Sagar telephoned to say that applications were not saving. The team’s first reaction was that it was one college’s internet, because the monitoring said the site had been up continuously for eleven weeks.

The monitoring said that because of one route:

app.get("/health", (req, res) => res.json({ ok: true }));

An external uptime service had been polling it every five minutes since launch. It returns two hundred as long as the process is alive and the HTTP layer works. It had been returning two hundred all week from four instances, none of which could reach the database.

The cause was found in about twenty minutes and was old and dull. A staff member had opened a psql session on Monday evening to check a count, run BEGIN, and gone home. The transaction stayed open. Autovacuum could not clean the applications table, which grew and slowed, and a long-running report query took connections and did not give them back. By Tuesday afternoon the pool was exhausted. Every write timed out. The homepage, which is static and served from a CDN, stayed up throughout, which is what everybody looked at.

The log volume was another part of the trap. Submissions were failing with a connection timeout, which was logged, but the same message had been appearing a few times a day for months, from a retry that always succeeded on the second attempt. Nobody had demoted it, so an error that fired forty thousand times on Wednesday looked exactly like an error that had always been there.

The count, once somebody wrote the query, was 11,400 failed submissions over three and a half days, from about 6,000 distinct students, none of whom had complained to anybody at the department. They had assumed the form was hard.

That set up the decision the team argued about on Friday afternoon, once the immediate fault was fixed. The obvious fix is to make the health check tell the truth by querying the database. But the platform is configured so that a failing health check restarts the instance. Four instances polling a struggling database every five seconds, then all restarting together and reconnecting at once, is a way of finishing off a database that was merely unwell, and every one of them would then fail the next check and restart again.

Against that: a check that returns two hundred from a process that cannot serve a single request had just cost six thousand students an application window.

There was a second decision, and it belonged to the department rather than the team: whether to extend the deadline. Extending is not free. The scrutiny committee’s dates are fixed, the disbursal is tied to the term, and every previous extension had produced a fresh set of last-minute applications and a complaint from the colleges that had made their students meet the original date.

Meera sells brass and blue pottery to buyers abroad. The site is a catalogue with an enquiry form, and one feature she is proud of: a box where a visitor pastes a few specifications and gets a product paragraph written for them, in English, by a model API. It was meant for her own use, writing listings. She left it open on the site because it was interesting, and because visitors who used it were about three times as likely to send an enquiry.

The route required no account. It took a text field and called the paid API.

For five weeks it was used thirty or forty times a day and cost around nine hundred rupees a month. On the ninth of October the provider’s dashboard projected forty-one thousand rupees for the month. The budget alert she had set at five thousand had arrived fourteen hours earlier, at two in the morning, in an inbox she reads once a day.

The logs told a plain story. From the fourth of October, a steady eleven requests a second, from a rotating set of addresses, each sending a prompt that had nothing to do with pottery. An unauthenticated route that calls a paid API is a free paid API for the entire internet, and it will be found by scanners within days of going live. Hers had been live for five weeks.

While she was in the invoice she found a second line she had never read. Egress: two point one terabytes. The catalogue’s hero image was a four megabyte photograph of a haveli courtyard, served from the origin at full size to every visitor on every page, and October’s traffic was a festival campaign.

So two costs, arriving together, in the fortnight that carries most of the year’s revenue. They are worth separating, because they have nothing in common except the invoice. The egress is a consequence of a file being larger than it needs to be, served from further away than it needs to be, and it will keep costing money in proportion to her success. The model spend is somebody else using a paid endpoint she left open, and it is not proportional to anything: it is whatever the script decides to do next.

She had also, without meaning to, given the abuser a discount. There was no maximum output length on the request, so every call generated as much text as the model would produce, and output tokens are priced at three to five times input tokens on the plan she was on. A cap of two hundred tokens would not have stopped the abuse and would have cut its cost by most of it.

The decision she had to make that morning was about the description box, and it was not obvious.

Turn it off now. The meter stops within a minute. It also removes the feature during the highest-converting fortnight of her year, on a site whose whole business is enquiries from strangers.

Leave it up and add a limit per IP address today. It is an hour of work. It is also weak in both directions: whoever is doing this is rotating addresses for a fraction of a paisa each, and a college or an office in a country she sells to shares one address between hundreds of people, so a tight per-address limit refuses a crowd of real buyers to inconvenience one script.

Leave it up and build the real control. An account, email verification, a daily quota per person, a hard cap on output length, a monthly ceiling at the provider. Two evenings of work, during which the meter keeps running at roughly four and a half thousand rupees a day.

The service books video consultations between village clinics and doctors in Nagpur. Monday between nine and eleven is the busiest window of the week by a distance, because that is when the clinics open and the queue outside is already formed.

The deploy went out at 08:40. It contained a new table, booking_slots, and code that writes a row to it for every booking as well as to the old bookings table, plus an SMS confirmation to the patient’s phone. It had been tested for a fortnight on staging.

At 09:05 the alert fired: more than a third of requests to the booking route returning 5xx, sustained for four minutes. The maintainer, who is one person with a laptop in Nagpur, opened the logs.

What he found first was not a cause. It was an arithmetic problem. Between 08:40 and 09:05, about sixty bookings had succeeded. Each of those had written a row into booking_slots (a table the previous build knows nothing about) and each patient had received a confirmation message telling them to come at a stated time. Rolling back would make those sixty bookings invisible to the application. Sixty people would arrive at four clinics, holding a message from a system with no record of them.

The error itself gave no hint. The stack trace ended in the HTTP client. Latency on the booking route had gone from two hundred milliseconds to thirty seconds. Requests to routes that had nothing to do with booking were also failing, which is the shape of a resource problem rather than a bug in one handler: something is holding workers, and everything queues behind it.

The scope told him a little more, and took a minute to establish. It was all four clinics, not one, so it was not somebody’s connection. It was every route rather than one, which pointed away from the code he had just changed and towards something shared. And it had begun before the alert: the error rate graph, read backwards, rises out of the noise at about 08:52, thirteen minutes before anything told him.

What he did not have was a dependency screen. The service calls three external things (the video provider, the SMS provider and the mail relay) and none of them had a rate or a latency graph. “Is it us or them” is the second question in most incidents, and at 09:06 he had no way to answer it in under ten minutes.

So at 09:07 there were two paths.

Roll back now. The fire stops. The evidence (logs, metrics, the bad artifact) all survives a rollback, so diagnosis loses nothing. The price is sixty invisible bookings, which is not an abstraction: it is somebody telephoning four receptionists and getting names written into a paper book before the patients arrive, and it is a repair job to write afterwards.

Spend ten more minutes understanding it. If it is something small and obvious, the fix is forward and nothing is lost. If it is not, thirty-eight per cent of the busiest two hours of the week continue to fail while he reads.

There was a third question, smaller and easy to skip: whether to say anything publicly before knowing what was wrong. The service has a status page. The instinct is to wait until there is something definite to say.

The college has about four thousand students and three counsellors. Before the model there was a list, maintained by a clerk, of students who had missed a lot of classes or failed papers. The clerk was reassigned in June and the list stopped.

The IT department built a replacement. Five years of records, a gradient-boosted tree, an area under the curve of 0.89 on a held-out set that had been kept properly separate. It produces a weekly roster of forty names, ranked. It went live in July.

By the end of August the head of counselling had a complaint that was hard to argue with. The roster kept naming students who, on meeting them, were plainly fine: a topper, twice. And three students had left in that period, none of whom had appeared on any roster.

Nothing in the system was broken. There were no errors, latency was flat, the model file was the one that had been evaluated, and the tests passed. The head of IT spent a day on it and found that the numbers going into the model in production were not the numbers it had been trained on. Three of them, specifically.

days_since_last_login was computed in the training notebook from timestamps in UTC and in the web application from timestamps in IST. For any student whose last activity was in the evening, which is most of them, the two paths differed by one. A feature the model leaned on heavily was systematically shifted.

attendance_pct came, in training, from a warehouse table that a nightly job cleans and back-fills. At serving time it came from the live row, where first-year students have a null until that job has run. Serving replaced the null with zero. Training had replaced it with the column mean. Zero means “never attended”, which is the strongest signal the model has, and every first-year student was being handed it.

programme was normalised in the training pipeline (trimmed and uppercased) and not in the serving path. So “B.Sc “ and “b.sc” arrived as categories the encoder had never seen, and the encoder mapped them to index zero, which is the most common category. Not an error. A confident, quiet lie.

The decision that had to be made was not technical, and it belonged to the head of counselling. The fix would take two to three weeks.

Suspend the model. Then the counsellors need a list from somewhere, and the clerk who used to make one is in another department. Rebuilding it by hand costs about a fortnight of the counsellors’ own time, which is the exact resource the model was bought to protect, in a term that has already started.

Keep using the roster while it is fixed. It is cheap and it is already in everybody’s calendar. It is also close to random, and a name on that list receives a counsellor’s hour that a different student does not receive. Three weeks of that is three weeks of misallocated attention with a false claim of rigour attached to it.

The school takes fees online from about nineteen hundred families. Fee week runs Monday to Friday, with a late fee after that, and the portal is the only online route: the alternative is a counter in the office that manages perhaps two hundred families a day.

On Tuesday afternoon a student in class nine looked at the address of his own receipt, /receipt/4411, and typed 4412. He got a receipt belonging to a family he did not know: the parent’s name, their mobile number, the amount paid, the date, and the last four digits of the card. He showed his class teacher, who took him to the office, who telephoned the vendor who built the portal.

The portal checks that somebody is logged in. A middleware does it, on every route, correctly. What it does not check is whether the logged-in parent has anything to do with the receipt being requested. The handler loads the row by ID and renders it. This is broken object-level authorisation, and it is consistently reported as the most common serious flaw in web APIs, largely because nothing about the request is invalid. No tool is required. Somebody changed a number.

The fix is small and everybody could see it: put the ownership into the query, so the row is fetched by ID and account together and a forbidden row and a missing row take the same code path. But the same pattern appears in four places: receipts, transport records, the term invoice, and a bulk endpoint that takes a list of IDs and, in the version written last year, checks the first one.

The vendor found the fourth of those by doing what the lesson recommends and nobody does: opening one object he owned, noting the identifier, and requesting the neighbouring identifier from a second account. Fifteen minutes with a browser and two logins found three of the four holes. The fourth needed a grep for every place a record is loaded by identifier, which produced eleven call sites, of which four went straight to rendering.

One more thing was true and made the week worse. Nineteen hundred receipts is nineteen hundred parents’ names, mobile numbers and payment amounts, walkable in a loop by anybody with an account, and any parent at the school can get an account, because that is what the portal is for. The exposure was not one receipt. It was the fee register of the whole school, available to every family in it.

That is where the decision sat, on Tuesday evening, with the late-fee deadline on Friday.

Take the portal offline until all four are fixed and tested. Nothing further leaks. Nineteen hundred families lose the online route for at least a day, the counter cannot clear the backlog before Friday, and the school would have to waive the late fee for everybody, which the trustees would have to approve and which sets a precedent they had refused twice before.

Patch it live, during fee week. The change is a few lines per handler. It is also a rushed change to the code path that takes money, in the week that takes most of the year’s money, and a mistake there is worse than the leak.

A second decision was harder and nobody wanted it. The school did not know whether anybody had done this before the student did. The access logs would say, and they roll after fourteen days.

The trust runs a lending library for about nine hundred active borrowers, with three thousand one hundred borrower records and nine thousand titles. The site was built in 2023 by a volunteer, Anoop, who did it well and then took a job in Bengaluru in January. Nobody has deployed anything since.

In November two emails arrived within a week. The managed database provider was discontinuing the free tier the site runs on, in thirty days. And the language runtime the application uses reaches end of life in February, after which it stops receiving security patches: a date that had been published three years in advance.

The trust’s position when it looked at what it had:

• The repository exists and is complete. The README has four steps. Three of them fail on a machine that is not Anoop’s laptop.
• Nobody at the trust knows how to deploy it. There is a deploy script; it references an environment variable nobody can find the value of.
• The domain renews on a card belonging to Anoop’s father. Auto-renew is on. Nobody at the trust has access to the registrar account.
• There is a nightly database backup. Nobody has ever restored one.
• The librarian uses the lending screen for every issue and return, about seventy a day.

The trustees had thirty days and three options, each with a cost they could name.

Pay for the managed database and buy Anoop’s time. Roughly two thousand rupees a month for the database, plus a weekend or two of a former volunteer’s paid time for the runtime upgrade and a handover. It is money the trust does not have spare, and it puts them back where they started: dependent on one person who lives elsewhere.

Move everything onto a small virtual machine themselves. About five hundred rupees a month, a quarter of the price. It also means somebody at the trust owns an operating system: patches, firewall, backups, restores, a database to upgrade, and a person to be woken. There is no such person. The librarian is a librarian.

Retire the lending half and keep the catalogue as a static archive. Crawl the catalogue into flat HTML, host it free, serve 301s from the old catalogue URLs, export the lending records to CSV, and go back to the paper register the library used before 2023. The running cost falls to almost nothing and the links keep working. It also removes the one screen the librarian uses seventy times a day.

Before choosing, one trustee asked the question that should have been asked in 2023 and had not been: if we take the paid database plan, how do we leave it later. The answer was reassuring and it was not luck: the database is ordinary Postgres, the dump command is standard, and there is no export queue or support ticket between the trust and its own data. Had the volunteer chosen a proprietary store with a vendor-specific query language, the same thirty days would have been a rewrite rather than a bill.

The other quiet fact in the room was that the lending screen is the only part anybody uses daily, and it is also the part that holds state. The catalogue is a read-only view of a table and could live anywhere for nothing. Almost the whole of the trust’s operating burden came from one feature, which is a useful thing to know before deciding what to keep.