Knowing If It Works

Vidyarthi Classes has 1,400 students across three batches and a doubt-solving problem that every coaching centre has: between nine at night and seven in the morning, nobody answers. The founder, Rakesh, had been paying two junior teachers to sit on the WhatsApp doubt groups until midnight, and they were leaving.

Over the summer a former student, now in his second year at an engineering college, built a doubt-solver on top of a hosted language model. You send a photo or a typed question, it replies with a worked solution. Rakesh tried it himself. He typed fifteen questions from the printed module on kinematics and rotational motion, read the answers, and all fifteen were right. Two were better explained than the printed solutions. He wanted it in every group by Monday, when the new batch started, because a competitor two streets away had started advertising “24x7 AI doubt support”.

The head of physics, Mrs Iyer, had been teaching for twenty-two years and did not trust it. Not on principle: she had read three of the fifteen answers and thought they were fine. Her objection was that she had no idea what the other few thousand answers would look like. “The questions you typed are from our own module. Our students do not send those. They send a blurry photo of a question from a random book, with half the diagram cut off, and they write ‘sir why not option C’.”

Rakesh’s position was that fifteen for fifteen was evidence, and that the cost of waiting was real. Every night the tool was not live was a night the competitor’s advertisement was true and his was not. He estimated that three days of building a proper test would cost him the launch date, and that a fortnight of “it works, we tried it” was worth more than a number.

Mrs Iyer’s position was that fifteen for fifteen told them almost nothing. If the true failure rate were one in four, fifteen clean passes would still happen often enough that nobody should be surprised. She also pointed out that Rakesh was the worst possible reader of the outputs, because he knew what the answer should be and filled in the gaps as he read. And she had a specific fear: a solver that confidently gives a wrong numerical answer to a student at 1 a.m., who then goes into the test believing it.

The two of them agreed on one thing, which was that the argument was going in circles. So they wrote down what a decision would actually require.

The decision was not “launch or not”. It was whether the tool answered students directly in the groups, or whether it drafted an answer that a junior teacher approved in the morning. Mrs Iyer wanted a threshold: on a hundred real doubts from last year’s groups, the tool had to give a correct final answer on at least 85, and for numericals the answer had to be correct to the printed solution. The unacceptable failure was a confident wrong numerical answer: she wanted that scored separately, and she wanted a single occurrence in a hundred to fail the run.

Building the set was the cost. The doubt groups from last year were still on two teachers’ phones. Pulling a hundred real doubts, stratified so that photographed questions, typed questions, conceptual doubts and numericals were all represented, would take a day. Labelling them took two teachers, and the labels had to agree with each other, which meant a labelling guide and a check that the two teachers actually applied it the same way. Three days, two senior teachers, at the busiest time of the year.

Rakesh’s counter-offer was to launch in one batch on Monday and “watch it closely”. Mrs Iyer’s counter-offer was that watching it closely was what he had just done with fifteen questions.

What would you have decided, and what would you have measured first?

The hospital’s outpatient department had started taking symptom messages on WhatsApp during the pandemic and never stopped. By 2026 it received about 2,000 messages a week: a photo of a rash, “my father has had fever for three days”, a question about a prescription, a request to change an appointment. One nurse, Sunita, triaged them between her other duties and could realistically read about 250 in a working day.

A vendor offered a classifier. Each message got a score between 0 and 1 for “needs same-day clinical attention”, and the vendor’s deck said the model had an AUC of 0.91 on their evaluation set and an accuracy of 95%. The medical superintendent, Dr Pawar, was interested, and the hospital’s one IT person, Prakash, was asked to make it work.

Prakash did the thing this course spends a module on. He took 500 messages from the previous two months, had Sunita and a junior doctor label each one as urgent or not, and found that 6% were genuinely urgent. Then he ran the vendor’s model over them and swept the threshold.

At the default of 0.5, the model flagged 90 of the 500. Of those, 21 were really urgent, so precision was 23%. It missed 9 of the 30 urgent messages, so recall was 70%. Scaled up to a week’s traffic of 2,000, that is 360 flagged messages, which Sunita could read, and roughly 36 urgent messages missed every week.

At 0.3, the model flagged 150 of the 500 and caught 28 of the 30 urgent ones. Recall 93%. But 150 of 500 is 30% of traffic, which is 600 flagged messages a week for a nurse who can read 250. The queue would be two and a half days behind by Wednesday, and an urgent message at the back of a queue is a missed message with extra steps.

At 0.8, the model flagged 24 and caught 16. Precision 67%, recall 53%. Almost half the urgent messages would go unflagged.

The vendor’s accuracy figure was true and useless. A model that flagged nothing would score 94% accuracy on this data, because 94% of messages are not urgent.

Dr Pawar wanted the highest recall available. A missed urgent message was, in his words, the only failure that could end up in a newspaper. Sunita pointed out that the highest recall available would bury her, and that a flag she could not get to was worth nothing. Prakash had a third concern that neither of them had raised: the model’s scores did not mean what they said. When he bucketed the 500 messages by score, the ones scored between 0.8 and 0.9 were urgent about half the time, not 85% of the time. The vendor’s threshold advice, which assumed the score was a probability, was built on sand.

The decision had two parts. Where to put the line, given that every position traded a missed patient against a queue the nurse could not clear. And whether to trust a single line at all.

There was a budget question underneath it. The hospital could not hire a second nurse for this. It could, possibly, get a junior doctor to spend forty-five minutes a day on the queue. Forty-five minutes was perhaps another 80 messages.

How would you set the threshold, and what would you tell Dr Pawar about the 95% accuracy in the vendor’s deck?

Kavitha runs an export business with her brother that makes knitted childrenswear for four buyers in Germany and the Netherlands. The buyers write long emails in English and occasionally German: order changes, delivery dates, fabric specifications, complaints. The production supervisor reads Tamil. For years Kavitha translated by hand, in the evenings.

Her brother Arun, who handles the accounts and is the closer thing to a technical person, built a prompt that turned each email into a Tamil summary with a fixed structure: what changed, the new deadline, the quantities, anything the supervisor must act on today. It worked well enough that Kavitha stopped translating. Then a shipment went out with the old collar specification, because the summary had dropped one sentence from a German email, and the buyer deducted 4% from the invoice.

Arun rewrote the prompt. He also, at Kavitha’s insistence, built a test set: a hundred real emails from the last two years, each with a hand-written Tamil summary and a short list of facts that had to be present. He ran the old prompt and the new prompt over all hundred. Old prompt, 82 correct. New prompt, 87.

Arun wanted to switch. Five points was five points, and the new prompt had been written specifically to stop the collar problem. Kavitha, who had spent an afternoon reading the module on error bars that her nephew had sent her, said that on a hundred items five points was inside the noise and asked him to prove it.

The argument had a shape worth pausing on. Arun’s cost of not switching was real: every week the old prompt ran was a week with the known failure in it. Kavitha’s cost of switching was also real: the summaries were now the only thing the supervisor read, and if the new prompt had regressed on something else, deadlines for instance, nobody would notice until a container was late.

Arun’s first move was one the course warns about. He re-ran the new prompt twice more, at the default temperature, and got 85 and 89. He proposed averaging the best two. Kavitha said that was the same as running until it looked good. She was right, and he knew it.

So he did the paired thing. Instead of comparing 82 against 87 he listed which items had changed verdict. Both prompts got 79 items right and 10 items wrong. On the remaining 11, the old prompt was right and the new one wrong on 3, and the old prompt was wrong and the new one right on 8. The difference in the totals was 5, which is 8 minus 3.

The rule of thumb from the course says the difference is worth believing when the absolute gap between the two flip counts exceeds twice the square root of their sum. Twice the square root of eleven is about 6.6. The gap was 5. Not conclusive. Close, but not conclusive, and a hundred items was what they had.

Arun’s suggestion was to add fifty more emails and re-run. Kavitha’s suggestion was to first look at the three items the new prompt had broken.

All three were deadline items. In each, the German email gave a date in the form 14.03., and the new prompt, which now insisted on quoting the changed specification verbatim, had summarised the specification beautifully and rendered the date as 3 April. The eight improvements were all specification items, which is what the rewrite had been for.

So the honest summary was not “the new prompt is five points better”. It was “the new prompt fixes specifications and breaks German dates, and on this set we cannot tell whether the net effect is positive”.

What should they ship, and how many more emails, if any, should they label?

The Thrissur District Co-operative Bank processes about 9,000 small loan applications in the eight weeks before Onam: gold loans, two-wheeler loans, festival advances. Each application is a bundle of paper: the form itself, an identity document, an address proof, and for salaried applicants a salary slip. Clerks in 34 branches type the fields into the core banking system by hand, and the backlog in the last fortnight of the season is the single biggest complaint the bank receives.

A vendor demonstrated a document pipeline: photograph the bundle, and the system classifies each page, reads it, and fills the form fields automatically. The demonstration was on the vendor’s own sample of forty documents and reported 98% field accuracy. The general manager wanted it in the branches by the first week of August. The head of operations, Mr Menon, had one question, which was how many applications would come through with every field right, because a form with one wrong field still has to be handled by a person, and if the person has to check every field to find the wrong one, the pipeline has saved nothing.

The vendor did not have that number. Neither did anybody else.

The bank’s IT officer, Deepa, was given the job of finding out, with a constraint: three weeks before the season, and every week of measurement was a week less of rollout. The vendor offered their sample set, already labelled, and the argument for using it was speed. Deepa’s argument against was that the vendor’s forty documents were flat scans of clean forms. The bank’s intake was a branch clerk’s phone camera, at an angle, under a tube light, with a rubber stamp over the date and, in the gold-loan forms, a thumbprint where a signature should be.

The decision in front of Mr Menon was whether to accept the vendor’s measurement and roll out with a clerk checking a sample, or to spend a week building a set from real intake and measuring the things the vendor had not.

Deepa’s proposal was specific. Take 150 application bundles from last season, photographed the way clerks actually photograph them, from six branches including two rural ones. Label the fields by hand, which she estimated at two clerks for three days. Then measure three separate numbers, not one: whether the output was valid at all, the accuracy of each field, and the proportion of whole applications with every required field correct. And measure each stage of the pipeline separately, because the vendor’s system did three things in sequence, page classification, text extraction, and field extraction, and a wrong page classification made every downstream number meaningless for that bundle.

She also wanted to know how many distinct salary-slip layouts the 150 bundles contained, because if most of them came from three large employers the score would describe three templates and not the world.

The cost of Deepa’s week was a week. The cost of skipping it was that the bank would learn the whole-record number in the second week of the season, from the clerks, in the branches with the longest queues.

What would you have measured, and what would you have done if the whole-record number came back well below the field number?

Padhai Labs sells an essay-practice product to CBSE schools. A student writes an essay on the app, and an AI tutor returns feedback: what worked, three things to improve, and a corrected paragraph. Teachers liked it, sales were growing, and the product team wanted to ship changes to the feedback prompt every fortnight.

Reading the feedback by hand did not scale. Two thousand essays a week came through, and the product lead, Nikhil, wanted a number he could watch. So the team built a judge: a second model prompt that read the essay and the feedback and returned a score out of ten for “feedback quality”, with a sentence of reasoning. The average across a week was 8.4. Nikhil put it on the dashboard.

A new feedback prompt was written that produced longer, more structured feedback with headings and a summary line. On the judge, it scored 8.9 against the old prompt’s 8.4. Nikhil wanted to ship it on Monday.

The evaluation engineer, Meera, said the judge had never been checked against a human and that until it had, 8.9 versus 8.4 was one model’s opinion of another model’s prose. She asked for a week to calibrate it: sixty essays with feedback, labelled independently by two English teachers on the staff, and the judge’s verdicts compared against theirs.

Nikhil’s objection was cost and time. Two teachers for five hours each was a real expense, the teachers had classes, and a week’s delay on a prompt that scored half a point higher looked like process for its own sake. “If the judge says it’s better and it reads better, what are we waiting for?”

Meera’s answer was that it read longer, and she did not know whether it read better, and neither did he. She also had a specific worry. The judge scored on a 1-to-10 scale, and she had noticed that almost every score was a 7, 8 or 9. A scale where everything lands in the same three numbers carries almost no information, and a model asked to compare two pieces of feedback tends to prefer the longer, more structured one whatever its content.

The decision was whether to ship the new prompt on the judge’s say-so, or to spend the teachers’ ten hours first.

Meera also pointed out that the judge and the feedback generator were the same model family, and that models tend to prefer their own family’s prose. That was one more reason to doubt the half point, and one more thing a calibration set would settle.

There was a second question underneath, which Meera raised and Nikhil initially waved away: what the score was for. Nobody could say what a move from 8.4 to 8.9 meant in terms of a student’s writing. What the teachers actually cared about, when asked, was three specific things. Did the feedback misstate what the student had written? Did it correct a sentence that was already correct? Did the three suggestions address the essay’s actual weaknesses rather than generic advice? Each of those is a yes-or-no question about an observable thing, and none of them is a 1-to-10.

What would you have done with the judge, and would you have shipped the new prompt on Monday?

Netra Support runs the WhatsApp assistant for a regional broadband company with about 300,000 subscribers. The assistant answers billing questions, books technician visits and explains plan changes, and it hands over to a human agent when it cannot. The team of six had done the early work properly: a 200-item eval set from real chats, a rubric, a judge, and a CI job that ran the set on every pull request and failed the build if the pass rate fell below 85%.

Within a month the gate was a joke. The model was stochastic, so the score moved two or three points between identical runs. A pull request that changed only the README went red. A developer added an automatic retry. Another raised the threshold to 83. By the time the founder, Sameer, noticed, the job had been marked “allow failure” and nobody had looked at its output for three weeks.

Then a customer was told the wrong window for a refund, twice, and posted the screenshots.

Two things were in front of the team at the same time. The first was what to do about the gate. The engineering lead, Farah, wanted a week to restructure it into tiers: deterministic checks that blocked, a small core set that blocked only when the drop exceeded a band derived from the measured run-to-run spread, and the full set nightly and non-blocking. Items that flipped between runs would be quarantined rather than retried. Sameer’s view was that a week of the lead engineer’s time on test infrastructure, in the fortnight after a public incident, was a week not spent on the incident.

The second was the A/B test. A rewritten reply prompt had been in a 50/50 experiment for four days of a planned fourteen, with resolution-without-handover as the primary metric and a pre-registered minimum effect of three points. On day four it was six points ahead with a p-value of 0.03. Sameer wanted to ship it that afternoon, partly because it would be something positive to say publicly. Farah said the p-value assumed one look at the end and that a daily-checked fortnight ran a false-positive rate nearer a quarter than a twentieth. Sameer said he did not care about the statistics if the customers were happier.

Farah did one more thing before the argument reached a decision. She and a colleague took fifty failed items from the last full run and read them, writing one line each about why, before assigning any category. Twenty-one of the fifty were the assistant quoting a refund window that had been correct until the company changed its policy in June; the knowledge base article had been updated, the retrieval index had not. Eleven were the eval set’s expected answer being wrong for the same reason. Nine were the assistant answering in English when the customer had written in Telugu. Six were genuine model failures on the prompt. Three were miscellaneous.

So the incident had not been a prompt problem, and the A/B was comparing two prompts on top of a stale index.

What would you ship, what would you fix first, and what would you do with the day-four result?

The department administers a post-matric scholarship that receives about 400,000 applications a year. Each application carries income and caste certificates, and a share of them are flagged for re-verification by a district clerk before payment. Historically the flagging was done by rule and by hand, it took three months, and students missed a semester’s fees waiting.

A vendor proposed a system that read each application and produced a flag, “needs document re-check”, with a confidence score. The proposal came with a system card: 96% accuracy on the vendor’s evaluation set, an ISO management-system certificate, and a line saying that every flagged application would be reviewed by a human before any action.

The joint secretary in charge, Mrs Rathore, was under pressure to sign. The scheme opened in six weeks, the vendor was ready, and the accuracy figure had already been quoted in a briefing to the minister. Her deputy, Anand, had two objections and a proposal.

The first objection was that 96% was measured by the vendor, on a set the vendor built, with a metric the vendor chose. Nothing on the card said what the set contained, how many items, what the base rate of genuinely problematic applications was, or whether the figure held for applications in Urdu, which were about a fifth of the total in three districts.

The second was the human review line. Anand had visited the district office in Bahraich. The clerks would receive flagged applications in a list of forty and approve them with one button. Nobody had measured how many wrong flags a clerk actually caught in that interface, and Anand’s guess, from watching for an hour, was very few.

His proposal was to spend three of the six weeks measuring. Take 300 applications from last year, stratified by district and by language, whose outcomes were known: which ones had genuinely needed re-checking. Run the vendor’s system over them and compute precision and recall per slice, with intervals. Then, separately, seed a small number of known-wrong flags into a test queue and measure how many the clerks caught, telling the clerks in writing that seeded cases existed and that the results would size the system and never be used against an individual.

Mrs Rathore’s cost was straightforward. Three weeks of a six-week runway, the clerks’ time, and the political embarrassment of a minister’s number being revised downwards in public. The vendor pointed out, accurately, that their certification covered their process and that no department had previously asked for anything of the kind.

Anand’s cost, if the measurement was skipped, was that the first anybody would know about a per-language gap would be when Urdu-medium students in three districts were flagged at twice the rate of everyone else, and the minister’s office found out from a newspaper.

There was also the question of what the confidence score meant. The vendor’s interface sorted the queue by it, so clerks would see the highest-scoring applications first, and nobody had checked whether an application scored 0.9 was actually more likely to need a re-check than one scored 0.6. If the score was decoration, the queue order was too.

Should she sign, measure, or both, and what would she need to see before the scheme opened?

PaisaSathi runs a lending app with about two million users and a support assistant that answers questions about EMI dates, statements, KYC status and repayment. The assistant had been stable for most of a year. The team had done the early work: a frozen eval set of 300 items sampled in January, a calibrated judge, a canary set of twenty items run hourly, and a pinned model version in a config file.

In the first week of September two things arrived within days of each other. The model provider sent a deprecation notice: the pinned version would be withdrawn in sixty days, and the recommended replacement was a newer model under a rolling alias. And the head of support, Priya, reported that complaints tagged “assistant unhelpful” had risen by 40% over six weeks, with nothing deployed on the assistant in that time.

The engineering lead, Rohan, had three candidate explanations and no evidence for any of them. The provider might have changed something under the pinned version despite the pin. The users might have changed. Or nothing had changed and the complaint tag was being used differently.

The decision that could not wait was what to do about the deprecation. The product manager wanted to switch to the new alias immediately: the provider’s announcement described it as better on every benchmark, the deprecation was going to force the move anyway, and moving now might fix the complaints. Rohan wanted to hold the pinned version, run the frozen set against both models, read the per-item diff by slice, re-tune the prompt for the new model, and then canary. He estimated four working days. The cost of his four days was four more days of rising complaints if the model was the cause. The cost of switching immediately was that whatever the new model did differently, nobody would know until users did.

The canary set answered the first question in an hour. Its twenty items had produced byte-identical outputs for eleven weeks. The pinned model had not changed. Whatever was driving the complaints, it was not the provider.

So Rohan looked at the inputs. He bucketed August’s traffic and the January eval set on language, intent and input length, and computed the total variation distance between the two mixes. It was 0.27. In January, Marathi-language messages had been 4% of traffic; in August they were 19%, following a marketing campaign in Maharashtra’s tier-two cities. The eval set contained twelve Marathi items. The assistant’s pass rate on those twelve was 58%, which nobody had looked at because twelve items in a slice of 300 barely moved the headline.

That reframed both problems. The complaints were most likely a mix shift onto a slice the assistant handled badly and the set could not see. And the deprecation decision now had a specific question attached: was the new model better or worse in Marathi, and at what latency and cost?

The frozen set was the only asset that could answer it, and it was stale. Refreshing it would break comparability with every number the team had reported since January. Not refreshing it meant deciding a model migration on a set that no longer resembled the users.

What order would you do things in, and what would you report to Priya?