AI, Actually Explained

The district had a number nobody liked. Of the children who entered Class 6 in its government schools, a little under a fifth were no longer in a classroom by Class 9, and the office found out about most of them months late, when a headmaster mentioned it.

Two vendors came with dashboards.

The first sold an early-warning system. It had been trained on five years of attendance registers, term marks and mid-day-meal records from an adjacent district, and it produced a risk score between 0 and 100 for every enrolled child, refreshed weekly. The sales engineer quoted 82 per cent accuracy on data the model had not seen. The price was fourteen lakh rupees for three years.

The second sold a flagging tool. It raised a flag when a child missed seven consecutive days, or when marks fell by two grades in a term, or when an older sibling had already left the school. Three rules, written by two retired headmasters, printed on one page in the manual. Four lakh rupees, and it claimed nothing at all.

The education officer, who had sat through a decade of software demonstrations, asked the same three questions of both. What goes in and what comes out? Where did the behaviour come from: did a person write the rules, or did they come from examples? And when it is wrong, who pays and who finds out?

The answers separated the products immediately, and not in the direction the price suggested. The second product’s behaviour could be printed, argued with by a parent and amended by the office in an afternoon. It would also only ever catch what two retired headmasters had thought of. The first product could in principle find children the rules missed (the ones whose attendance is fine right up until it is not) and nobody in the district, including the vendor’s engineer, could say why any particular child had scored 78.

The officer pressed on one more thing. What is the score trained to predict? The engineer said children at risk of dropping out. The officer asked what the recorded right answer had been during training. The engineer, to his credit, answered honestly: children who did drop out, in the neighbouring district, in the five years on file. So the score predicted resemblance to children the schooling system had already lost, in a district with different schools, different distances and a different sugar harvest.

That is not a criticism of the vendor, who had built the only thing that could be built from the only records that exist. It is a fact about what the records are. A child who stopped attending and whom nobody entered as having stopped attending is not in the training answers at all, so the model cannot learn that such children exist. The office’s own registers had, by its own audit two years earlier, about eleven per cent of transfer certificates missing.

The officer also asked what the 82 per cent was 82 per cent of. It was the share of children the model classified correctly at the vendor’s chosen cut-off. Since only about a fifth of children leave, a system that predicted nobody would leave would score around 80 per cent on the same measure and be useless. The engineer knew this and had a better figure (of the children the model flagged, roughly one in five did in fact leave) which nobody had put on the slide because it sounds worse and means more.

A headmaster in the room asked the question that decided the meeting. Would teachers see the number? Because a teacher who sees 78 beside a child’s name treats that child differently, and there is no way to un-see it, and nobody would ever know whether the score had caused the thing it predicted.

Against all of this sat the arithmetic. Fourteen lakh was also bicycle vouchers for about nine hundred girls, which is an intervention with evidence behind it. And a wrong choice was not cheap in either direction: buy the rules and the children the rules cannot see stay invisible; buy the model and the office owns a number it cannot explain, attached to a child, for three years.

The bank has forty branches and about four lakh card transactions a month. Two years ago it bought a fraud-scoring service; the vendor’s model returns a number for every transaction, and the bank chooses the line above which the card is declined and the account flagged. The vendor shipped a default line and nobody had moved it since.

The risk officer had the season’s arithmetic in front of her. Roughly one transaction in two thousand turns out to be fraudulent, so about two hundred a month. At the current line, the model catches around 170 of them and wrongly flags about eight-tenths of one per cent of everything else: a little over three thousand two hundred genuine transactions. Nineteen false alarms for every real fraud, from a service the vendor’s brochure describes as 99.2 per cent accurate. Both numbers are true and only one of them is about the customer’s afternoon.

The season made it sharper. When the cane payments land, members travel: to Pune for a wedding purchase, to Mumbai for a medical appointment, to a district town for a tractor part. Spending patterns that the model has learned as normal stop being normal for six weeks, and declines rise exactly when a member most needs the card to work. Last year the branch managers logged four hundred and ten complaints in that window, and the bank’s members are also its owners, which changes what a complaint is.

Two levers were on the table. Raise the line for six weeks: fewer declines, and the finance committee’s estimate was around six lakh rupees of additional losses. Or hold the line: protect the money and accept four thousand-odd extra declines in the one period that generates the year’s goodwill.

There was a third consideration that the vendor raised and the board had not thought about. The model had last been retrained fourteen months earlier. Fraud is an adversarial problem: the moment a pattern stops working, whoever is running it changes the pattern, so a fraud model that is not being retrained is quietly getting worse while its published accuracy stays where it was. Two of the bank’s recent losses had involved a card-not-present pattern that did not exist when the model was fitted. That is an argument for spending money on retraining rather than for moving a line, and it was not on the agenda.

While the officer was preparing the note, a branch manager mentioned something that turned out to matter more than either lever. At two branches a flag was being treated as a finding rather than as a trigger for review. One member’s card had been blocked for eleven days pending an internal investigation that consisted of a form sitting in a tray. He had driven to the branch twice.

That detail reframed the decision. The threshold sets how often the system is wrong. It does not set what happens next, and what happens next was where the harm actually lived. A false alarm that is resolved by a thirty-second message is an inconvenience. The same false alarm, treated as a verdict with the burden of disproof on the member, is eleven days without a card.

Against the cheap fix sat a real cost. Confirmation messages require the member’s mobile number to be current, and about a fifth of the bank’s records were not. Updating them meant branch staff time in the busiest six weeks of the year, and the officer had no way to fund it except by taking staff off the counter. A confirmation that never arrives is worse than no confirmation, because the member is now blocked and has also been told, by the absence of a message, that the bank did not try.

The officer wrote the note with both options costed and a third section she had not been asked for, headed what a flag means. The board’s discussion spent eleven minutes on the threshold and fifty on the third section, which is the correct proportion and not the one anybody expected.

Fourteen hundred students, eleven faculty, and a doubt queue that does not respect office hours. The centre’s physics lead had spent three weekends wiring a chatbot to a WhatsApp number, and it worked well enough that the director wanted it live before the exam season.

The first version had pasted the centre’s own study module into the system prompt. That failed immediately: nine hundred pages does not fit on the desk, and what did fit was silently truncated from the top, so the bot was confident about thermodynamics and had never heard of the first four chapters. He rebuilt it to retrieve the four or five most relevant passages for each question and put only those into the context.

Then the real problems began, and every one of them was a property of the mechanism rather than a bug.

Students asked in Hinglish, which the bot handled better than anyone expected and slightly worse than it appeared to. Two students asked the same question about rotational inertia four minutes apart and got answers that differed in emphasis; they screenshotted both and posted them in the batch group with the caption that the bot did not know. It was, in fact, the sampler, and both answers were correct, but nothing in the interface said so, and a student who cannot check has been given a reason to distrust the tool at the moment they most need it.

Worse, the bot cited page 212 of Module 3 for a formula. Page 212 of Module 3 is a set of solved examples on capacitors. The model had produced the shape of a citation, which it has read ten thousand of, and the shape had a page number in it because page numbers are what goes there.

And a student asked how many questions from optics had come in the last three years. The bot answered with a confident distribution that belonged to a syllabus pattern from before the last revision, delivered in the same tone as everything else.

The physics lead noticed the pattern in his own failures once he stopped looking at them one at a time. Everything the bot got right was work on a passage that was in front of it: rearranging an explanation, working a numerical example, restating a definition in Hinglish. Everything it got wrong was something it had to remember: a page number, an exam pattern, a date, a marks distribution. The two categories were not equally hard to a student and were entirely different to the machine, and no amount of instruction moved a question from the second category to the first. Only handing it the material did that.

He also learned something about his own prompt. It had grown across three weekends into two thousand eight hundred words of accumulated rules: be encouraging, be brief, use SI units, never give a full solution, always give a full solution when asked twice, address the student by name, reply in the language of the question. Several of these contradicted each other, and the bot resolved the contradictions differently on different days, which is how a set of instructions becomes indistinguishable from randomness.

The director’s position was that the bot goes live. The 1 a.m. doubts do not disappear when the centre closes; they go to a Telegram group run by nobody, where the answers are worse and unattributable, and the centre’s competitors had launched something similar in March.

The physics lead’s position was that a wrong answer from the centre’s own number, eight weeks before the exam, in the voice of the institution the parents had paid, is a different object from a wrong answer in a Telegram group. The centre carries that one.

The lecturer had heard about the detectors in a staffroom conversation and found one that was free and needed no account. She pasted the essays in one at a time over an evening. Twenty-six came back above eighty per cent likely to be AI-generated. Four were above ninety-five.

She took the list to the head of department, and the first thing anyone noticed in that meeting was that twenty-one of the twenty-six students had come through Tamil-medium schooling. The highest score in the batch, ninety-six per cent, belonged to a student who had spent the previous year at a spoken-English coaching class and wrote exactly the way it had taught her: short declarative sentences, a small and repeated vocabulary, the same three connectives.

The college’s regulations allowed the head of department to cancel an internal assessment on his own finding. He had, in principle, everything he needed.

What he did not have was any published false-positive rate for the detector on Indian English, and when a colleague went looking for one, she found instead a 2023 study in Patterns that had run essays by non-native English writers through seven detectors and had more than half of them misclassified as generated, while essays by native speakers were classified correctly far more often. The mechanism was uncomfortably close to the room: detectors flag low variation in word choice and sentence structure, which is precisely what a second-language writer produces, and precisely what the coaching class had trained.

Two other facts complicated the meeting.

One student, unprompted, admitted that he had used a chatbot for most of his essay. His score was thirty-four per cent.

And a second student, whose essay was entirely her own and who had been flagged at eighty-eight, sent a message saying that everything gets called AI now and that she would not attend the committee. She was not being difficult. She had correctly identified that the accusation cost her nothing to receive and cost the college nothing to make, and that she had no way to prove a negative.

A third fact emerged when somebody thought to test the tool rather than to argue about it. A lecturer pasted in two paragraphs of a Tamil Nadu board textbook, printed in 2011, and got seventy-nine per cent. She pasted in a paragraph of her own doctoral thesis from 2009 and got ninety-one. Neither could have been generated by anything, because nothing existed to generate them, and the tool had no way to say so and no interval around its number. It returned a percentage with two decimal places.

That is worth sitting with, because a percentage with two decimal places is a claim of precision that the underlying method cannot support, and it is the reason the number was persuasive in the first meeting. Nobody in the room had asked what the figure was a percentage of. It is not a probability that the essay was generated; it is a score the tool computes from how little the text varies, mapped onto a scale by the tool’s own authors.

The costs were real on both sides and the head of department said so out loud. If generated essays pass, the honest students are cheated, the department’s internal assessment stops meaning anything, and the staff who mark carefully are wasting their evenings. If the detector’s number is treated as evidence, the college will discipline twenty-one Tamil-medium students for the way they write English, in a year when the department had been trying to persuade exactly those students that they belonged there.

The clinic sees about three hundred live matters. Its files arrive as scanned Marathi documents: rent receipts, muster rolls, notices, handwritten acknowledgements. Two volunteers spend most of their week reading them and typing dates, amounts and party names into a spreadsheet before any lawyer sees the case.

The team had a genuine architectural choice and eighteen lakh rupees over two years to make it with.

The closed route was a hosted frontier model behind an API. It handled Marathi better than any open model they tested on day one, needed no hardware, and worked that afternoon. Against that: the clinic’s documents carry named parties, and sending them to a foreign provider is a disclosure the clinic would have to justify to the people whose disputes they are. The model could be changed or retired on the provider’s schedule, which the team had already experienced once on a previous project when an output format shifted overnight. And the per-document price was set by somebody else.

The open route was a small open-weights model, quantised, running on hardware the clinic owned, with the Marathi work leaning on the openly released translation models from IIT Madras. Nothing would leave the building. The file they downloaded would be the file they ran in 2029. Against that: on the first week’s tests it was visibly worse, it needed either a graphics card at about two and a half lakh or rented time, and the card they had specified had moved forty per cent in price in four months for reasons that had nothing to do with them and everything to do with export rules and demand.

The team’s instinct was to argue the architecture. The clinic’s coordinator asked a better question, which was what the volunteers actually did all day. The answer was not summarising. It was extraction: pulling seven fields out of a scan and getting them right, three hundred times a month, where a wrong date is a missed limitation period.

That reframed everything, because extraction from a document in front of the model is the reliable side of the mechanism, and summarising a case is not. It also lowered the bar the open model had to clear. A frontier model’s advantage is largest where the answer must come from its memory, and this task had no memory in it at all.

The honest cost of being wrong was not symmetrical. Building on the API and being right meant a working tool in six weeks. Building on open weights and being wrong meant six months spent, a card bought, and a clinic still typing.

There was one more consideration that the team had not expected to matter and which turned out to. The clinic’s coordinator asked what she would tell a client who asked where their eviction notice had been sent. Under the closed route the honest answer is a paragraph about a foreign provider, a retention period, a setting, and a court order in another country that could override the deletion promise. Under the open route it is one sentence: it stayed on that laptop. She said, reasonably, that she could not give a client the paragraph.

The team also priced the thing nobody prices. Adapting an open model to Marathi legal documents needs examples, and examples need somebody who reads Marathi legal documents to check them. That is the clinic’s volunteers, at the clinic’s expense, for weeks. The API needs none of that on day one, which is precisely why it is faster and precisely why it never becomes the clinic’s own asset.

The film had to play at two hundred and forty anganwadi centres and travel on WhatsApp, which meant short, in Marathi, and recognisable. The unit’s three previous campaigns had been narrated by Dr Kulkarni, a retired district health officer whose voice is known across the region from a decade of radio spots. In February he had a stroke and can no longer record.

A production vendor came with two offers.

The first was the voice. Forty seconds of an old radio spot was enough, the vendor said, to build a clone that would read any script the unit wrote, in Marathi, for nine thousand rupees. He demonstrated it on a phone in the meeting and the unit’s own staff could not tell.

The second was the pictures. Instead of a shoot at one lakh forty thousand (a day’s crew, a village street, a mother, a health worker, permissions) the vendor would generate the footage from a description. He showed samples: a lane in the rain, a clinic doorway, hands holding a vial.

The hands were wrong in three of the five clips. In one, the vial had no cap and then had one. The unit’s programme officer, who had spent nine years in those villages, said something more useful about the lane: it was not from here. The doors were wrong, the drainage was wrong, and the way the water sat on the road was wrong. She could not have specified it in advance and she recognised it instantly, which is exactly the failure that costs a health film its credibility with the audience it is for.

The voice was the harder question, because the voice worked.

Dr Kulkarni’s son said his father would have wanted the campaign to go out and offered his consent on his father’s behalf. The unit’s medical officer said, correctly, that a man who cannot now speak cannot consent to being made to speak, and that the reason the voice was worth using was precisely the reason it should not be synthesised: people in those villages would believe the film because they believed him.

Against that sat the drive. Two hundred and forty centres, a monsoon window that closes, and a budget that could not absorb both a studio narrator nobody knows and a location shoot. And the unit knew what happens to a health message that nobody trusts, because it had spent the previous year undoing a WhatsApp forward about the same vaccine.

The forward was the reason the labelling question came up at all. It had carried a photograph of a child, and when the unit tried to establish where the photograph came from it found the image on a stock site, taken in another country, eleven years old. Nobody had generated anything. The most common fake in the district was a real picture with a false caption, and a reverse image search had settled it in thirty seconds.

That left the unit with an awkward asymmetry it discussed for most of an afternoon. If it labels its own two generated shots, it has told the truth about a film nobody was going to doubt. It has not made the forwards any easier to check, because a forward carries no label and absence of a label means nothing. What it has done is establish its own practice, in public, before anybody asks, which the district information officer argued was the only kind of credibility that survives an argument.

The revised supply agreement arrived at nine in the evening on a Friday, with a request for signature within forty-eight hours. Thirty-four pages of English. Meena’s advocate was at a wedding in Erode until Monday afternoon.

Clause 19 was new and it was an indemnity. She could read the words and could not tell what they obliged her to do if a shipment was rejected at the port.

Her first instinct was the obvious one: paste the whole PDF into a free chatbot and ask what had changed since the last version. It would take ninety seconds.

Two things stopped her, and neither was squeamishness. The agreement itself contains a confidentiality clause covering its terms, and pasting it into a free consumer tier means disclosing it to a third party under terms that, by default, permit the text to be used in training the next model. Whatever the practical risk, she would have done the thing the document forbids, on the same evening she was being asked to sign it.

She tried a version of it anyway, on a general question about indemnities, and got something useful. Then she asked what the penalty cap was, and got a figure (a number, a currency, a clause reference) that is not in the agreement. She only noticed because she had the page open. The figure had the right shape and no source.

She tried again, differently. She asked it to explain what an indemnity clause generally obliges a supplier to do, without pasting anything at all, and got a clear, useful, checkable explanation of a concept that is written about in ten thousand places. That answer was worth having and she could have got it from any textbook; the difference was that she got it at nine forty on a Friday.

The distinction between those two questions is the whole of this case. What is an indemnity is dense in the material the model was trained on, general, and cheap to verify. What does clause 19 of this agreement cap my liability at is specific, private, appears nowhere in any training text, and is exactly the kind of thing that comes back as a number with the right shape and no source.

Her brother, working on the same problem in the next room and not talking to her about it, had by then pasted the entire agreement into a different free tool. He had also, without thinking about it, accepted the tool’s default terms four months earlier when he signed up to draft a marketing email.

The cost on both sides was real. Sign without understanding clause 19 and she is exposed on the one relationship the business cannot lose. Miss the deadline and she risks the relationship anyway, in a season when the buyer has alternatives in three countries. Wait for Monday and she has six hours, not forty-eight.

There was also a quieter cost that she named to her brother the following week. Twenty-two people’s wages depend on this buyer. A tool that produces a fluent, confident, well-formatted answer at ten at night to a person who is tired and frightened of a deadline is at its most persuasive exactly when it should be at its least trusted, and no amount of knowing that changes how it feels in the moment. She had believed the invented penalty figure for about forty seconds, and the only reason she stopped believing it was that the agreement happened to be open on the other screen.

For eleven years the firm has hired about a hundred and twenty engineering graduates every June. In the January planning meeting the delivery head proposed thirty.

His case was not speculative and he had numbers. Three of the firm’s four largest accounts had asked for lower rates at renewal. The coding assistant, in use for fourteen months, was drafting a large share of the routine work. And the firm had run its own small measurement, which is more than most firms do: on well-specified tickets in a common framework, the least experienced engineers were meaningfully faster with the tool. A graduate costs about six point two lakh fully loaded in the first year and bills nothing for roughly five months.

The same measurement had produced a second result that the delivery head reported honestly. On the legacy insurance codebase (nine hundred thousand lines, fifteen years old, three people who understand why the reconciliation module is written the way it is) the firm’s two most senior engineers were slower with the assistant than without it. They read plausible suggestions, decided they were wrong, and wrote their own. Both had told the delivery head, before the measurement, that the tool was helping them.

The HR head’s case was structural. All twelve of the firm’s current team leads came through this intake. Nobody in the room could describe another route by which the firm makes a person who can look at an account manager and say no. And the accounts that pay the best rates pay them for exactly that person.

She also mentioned the 2025 payroll study showing employment for twenty-two to twenty-five year olds falling in the most exposed occupations while their older colleagues held steady. The delivery head said, accurately, that it is one study, in one country, in years that also had a hiring correction and rising interest rates. Both of them were right, which is what made the meeting hard.

A third voice in the meeting was the account director for the insurance client, who made a point neither side had. The firm does not sell code. It sells somebody who will be answerable at half past eleven at night when a reconciliation run fails during a month-end close. That answerability is what the rate is for, it cannot be produced by a tool that cannot be sacked, and it is currently produced by a process that starts with a graduate fixing small bugs badly.

The delivery head accepted the point and returned a harder one. The rung is not disappearing because the firm chooses to remove it. It is disappearing because the work on that rung has stopped being worth what a client will pay for it, and no amount of internal conviction changes a rate card. If the firm wants juniors, it has to find something for them to do that a client will still pay for, or pay for their training out of its own margin and say so.

The arithmetic on the table: cutting to thirty saves about five point six crore over two years, against a risk that cannot be priced: that in 2032 the firm buys its team leads at a premium from firms that kept making them. Nobody in the room had lived through this before, and both of the people arguing said so.

The proposal was for forty acres, a phased build to about three hundred and fifty megawatts, and an investment figure of eight thousand crore rupees. The council had been told four thousand jobs. The figure in the developer’s own annexure was four thousand during construction and about a hundred and eighty permanent, which two councillors found on page sixty-one the night before the vote.

The two asks were specific. A grid connection larger than the town’s entire current draw, which the state utility would build. And a water allocation for cooling, from a municipal supply that had run on tankers for nine weeks the previous summer.

The cooling turned out to be the one genuine lever the council held. A site can be cooled evaporatively, which uses a great deal of water and less electricity, or with closed-loop chillers, which use very little water and more electricity and cost more to run. It is a choice made per site, and it is not a technical choice in a district on tankers. The developer’s engineer was straightforward about it: closed loop would raise operating cost, and the cost would be passed to tenants.

A councillor asked what the computers would be doing. Nobody could say. The developer explained, honestly, that it does not know either: tenants rent the capacity and decide what runs on it, and the mix between AI training, AI serving and ordinary hosting changes weekly. That answer landed badly and was correct.

A second fact was raised by the district’s own environment cell. The state’s grid is around seventy per cent coal, so the same computation carries several times the carbon here that it would in a hydro-rich region. That is not something the council can regulate. It is something the developer’s customers increasingly ask about, which made it a lever of a different kind.

Against refusal sat the ordinary calculus of every such vote. Two neighbouring districts were competing for the same investment and one of them would say yes. The substation and the road would be built either way only if the project came here. And a hundred and eighty permanent jobs is not four thousand, but in that town it is not nothing.

The council also had to decide what it was competent to decide. A councillor proposed a condition that the site not be used for artificial intelligence. The municipal solicitor advised against it in three sentences: the council has no power to regulate what a lawful tenant computes, no means of inspecting it, and no definition of the term that would survive a challenge. The condition would be unenforceable, and an unenforceable condition is worse than none, because it lets everybody claim the matter was dealt with.

What the council could regulate was a pipe, a meter and a land classification. That felt small in a debate about a technology and it was the only part of the debate the council was actually holding. The state utility would decide the grid connection. The chips would be bought under rules written in Washington and manufactured on an island the council had no view on. The tenants would be signed in another country.

One councillor put it plainly in the minute she asked to have recorded: we are not voting on artificial intelligence, we are voting on forty acres and a borewell, and the only honest thing we can do is get those two right.