AI, Safety and What Goes Wrong

In February 2022 Roberto Mata sued the airline Avianca. He said a metal serving cart had struck his left knee on a flight from El Salvador to John F. Kennedy Airport in New York. He filed in a New York state court. Avianca moved the case to the federal court for the Southern District of New York, because claims about international flights fall under a treaty, the Montreal Convention.

Two lawyers at the firm Levidow, Levidow & Oberman worked on it. Steven Schwartz did the research and the writing. His practice had always been in the New York state courts, and he was not admitted to practise in this federal court. So his colleague Peter LoDuca put his own name on the filings.

In January 2023 Avianca asked the judge, P. Kevin Castel, to dismiss the claim. The argument was simple. The Montreal Convention gives a passenger two years to bring a claim, and Mata had waited longer. To keep the case alive, Mata’s side needed court decisions showing that the clock had stopped, perhaps because Avianca had been through bankruptcy.

The firm’s research service had limited access to federal cases. Schwartz had heard about ChatGPT from press reports and from his family. He had never used it. He took it to be a kind of search engine. He asked it to argue that the airline’s bankruptcy paused the time limit. Then he asked it for case law, for specific holdings and for more cases. Each time it gave him what he asked for.

On 1 March 2023 the firm filed its answer to the motion. It cited and quoted decisions with names such as Varghese v. China Southern Airlines, each with a volume and page number in the Federal Reporter. LoDuca signed it without reading any of the cases. He relied on a colleague of more than twenty-five years.

On 15 March Avianca’s lawyers replied. They said they could not find most of the cases, and the few they could find did not say what the filing claimed. The judge’s own search failed too. In April the court ordered LoDuca to file copies of the decisions, and warned that failing to do so would end the case.

The decision

Put yourself at the firm in the middle of April 2023. The other side says the cases cannot be found. The court cannot find them either, and it wants copies. The only place the cases have ever appeared is a chat window.

There are two paths. The first is to tell the court now. The citations came from a chatbot, nobody at the firm can verify them, and the filing should be withdrawn. That is embarrassing. It weakens a client’s case that was already in trouble, and it invites a sanction.

The second is to go back to the tool, ask again, and file what it gives. The tool is fluent and specific. It supplies judges’ names and docket numbers. Asked directly whether Varghese is a real case, it says yes.

Before you open what happened, choose a path. Then name the human decision, at each step, that brought the firm to this point.

The group hires around 240 nurses a year and receives roughly 3,000 applications. Since 2024 a shortlisting tool has ranked applicants on qualifications, clinical placement history, employment gaps and a structured written response. It shortlists 900; a panel interviews them; 240 are hired.

Nobody has ever computed a selection rate by group, because the application form does not ask, and the HR head has always considered that a point in the group’s favour. Asking a nursing applicant her community is legally awkward, socially loaded, and exactly the kind of question the group has spent years not asking.

The question arrived from outside. A hospital in the group’s network won a contract with a European health-staffing partner, whose supplier questionnaire asked whether automated employment decision tools were audited for adverse impact and whether impact ratios were retained. The honest answer was no, and the honest reason was that the group had no group data at all.

What a first look showed

The HR head did the one thing that needed nobody’s sensitive data. She built forty paired applications (identical qualifications, identical placement history, identical written response) differing only in the applicant’s name and the district of the nursing college. The paired-input audit took a weekend.

The scores came back with a visible pattern. Applications naming colleges in three northern and eastern states scored on average 6.4 points lower on a 100-point scale than identical applications naming Tamil Nadu colleges, and the gap survived when she matched the colleges on affiliation and course length. The tool had learned, from four years of the group’s own hiring, that the group hires locally. It was describing the past accurately.

She then computed what she could from data she already held. Selection rates by nursing-college state: 31% for Tamil Nadu applicants, 18% for out-of-state. The ratio is 0.58, well under four-fifths. This is not a protected characteristic in Indian law, so it is not itself a legal finding, and it is correlated closely enough with community and language that she could not treat it as unrelated.

The decision

Option one: collect voluntary self-identification, held by a separate team, never visible to the shortlisting pipeline or the interview panel, used only for a quarterly aggregate table. This is what mature equal-opportunity monitoring looks like elsewhere. The costs are not hypothetical. Response rates will be partial and non-random: applicants who fear discrimination disclose least, which biases the measurement meant to protect them. The group would be creating a category of sensitive data it currently does not hold, which is a breach exposure and a DPDP obligation it does not have today. Two of the four unit directors said plainly that asking the question would be read as an intention to use the answer.

Option two: refuse to collect it and monitor the observable proxies instead: college state, language of the written response, whether the address is rural. Weaker evidence, unlikely to satisfy the European partner, available immediately using data already held, and it catches the large problems, which is what a 0.58 ratio is.

Option three: change the scoring instead of measuring it. Drop college identity from the model entirely. Cheap, immediate, and it removes the group’s ability to see whether the gap persists through some other variable, which it very likely will, because placement hospital and referee name encode the same thing.

The medical director wanted option three because it was the only one that could be done before the next intake. The HR head argued that a repair applied without measurement is a repair nobody can evaluate.

The draft is good. It is better organised than the junior’s usual work, the structure is clean, and it cites three judgments in support of the proposition that a delay in framing charges weighs in favour of bail in this class of offence. Two are High Court decisions and one is from the Supreme Court. Each carries a citation in the correct format, a bench composition, a paragraph number and a short quoted passage.

The partner has been reading about a case in another country where two lawyers were fined after filing a brief citing six judgments that did not exist. She asks the junior where the citations came from. He says he used an AI assistant to find authorities on the point and then wrote the argument himself. He did not open the judgments. He says the assistant gave the paragraph numbers, so it must have read them.

It is 4.20pm. The client has been in custody for eleven weeks. Filing tomorrow means a hearing on Thursday. Filing on Monday means a hearing the following Tuesday, six days more in custody for a man the practice believes should not be there.

What checking actually costs

The partner works out the arithmetic rather than the principle.

The Supreme Court citation is the cheapest to check. It has a reportable citation that either resolves or does not, on a free database, in about two minutes.

The two High Court citations are harder. One is from a bench and year where the free databases are patchy. Locating it properly could take twenty minutes each, and if the citation is subtly wrong (right court, wrong year, or a real judgment about a different question) the search takes longer than if it were plainly fabricated, because a plausible near-miss keeps producing candidates.

There is a fourth object in the draft that the partner counts separately: the quoted passage in each citation. A quotation is the highest-risk item in any document, because it is what a reader trusts most and checks least, and a judge reading a quoted line from her own court’s judgment will check it.

So the honest estimate is between forty minutes and two hours, and the upper end takes the filing past the point where the clerk can lodge it.

The decision

The junior proposes a middle path: verify the Supreme Court authority, which carries most of the weight, keep the two High Court citations with the hedge “it has also been held”, and file in the morning.

The partner does not like it and cannot immediately say why in a way that beats six days of custody. She works it through out loud. The hedge does not reduce the risk, because a citation offered to a court is offered as an authority whether or not the sentence around it is softened. The consequence if one is fabricated is not a correction; it is a bench that stops believing the practice, in a district with one bench and a long memory. And the person bearing that cost is not the junior or the partner. It is this client and the next eleven.

Against that, the partner also has to be honest that the base rate is not one. The junior is competent, the point of law is well settled, and the most likely outcome by a wide margin is that all three judgments are real. Verification is being bought against a probability, not a certainty, and the six days are certain.

The essay title was “a week in my family that was difficult”. It is a good prompt and it is why the problem exists. Four hundred and eighty children wrote about a parent losing work, a grandmother’s illness, a sibling’s arrest, a father’s drinking, a diagnosis nobody outside the house knows about. The essays carry names, class sections, the names of relatives, the name of the housing colony, and in several cases a hospital.

The teacher has nine days and 480 scripts, alongside 22 teaching periods a week. In the third evening she pasted eleven essays into a consumer chatbot on her own phone and asked for a paragraph of feedback on structure for each. The feedback was good. She marked eleven scripts in nineteen minutes.

She stopped on the twelfth because the essay described a family court matter and she found she did not want to paste it, which is a more reliable instrument than most policies.

She told the head teacher, which is the part of this case that made everything afterwards possible.

What was actually disclosed

The head teacher worked it through with the school’s IT coordinator rather than assuming.

Eleven essays left the phone over an encrypted connection and arrived at a provider’s servers in a region nobody chose. They were written to a request log before any model saw them. They sit in the account’s conversation history. They are very likely retained in an abuse-monitoring store with a different retention period and a different set of staff able to reach it. The account was a free consumer account, so the default was to train on the content, and the teacher had never visited the settings page.

Under India’s DPDP Act the threshold for a child is 18, which covers every one of these children, and processing their personal data requires verifiable parental consent. The consent that failed here was not one the teacher declined to obtain. It was one nobody was ever in a position to give: the children did not consent, the parents were not asked, and the act of pasting made the school the party determining the purpose of the processing.

The decision

The head teacher had a staff meeting in four days and three options, each with a cost.

Ban it. Simple to announce, impossible to enforce on personal phones, and the predictable effect is that the next teacher does not tell anyone. The school would trade a manageable exposure it can see for an invisible one, and it would lose the only reason it knows about this incident at all.

Buy an approved tool. A business-tier account with training contractually excluded, bounded retention and a data processing agreement costs money the school has, budgeted this year for six laptops for the computer lab. It also does not solve the underlying problem, because a compliant cloud tool still means 480 children’s family circumstances leave the building.

Put a local model on a school machine. A 4-bit quantised model of seven to nine billion parameters needs about five gigabytes and runs on the 16 GB staffroom desktop the school already owns, through free software. Nothing is transmitted. It is weaker at hard reasoning and entirely adequate for the task actually in front of the teacher, which is structural feedback on a 300-word essay. It costs an afternoon of the IT coordinator’s time, produces five to fifteen words a second, and requires teachers to walk to the staffroom.

The deputy head’s objection was the sharp one: a tool that requires walking to the staffroom at 9pm will lose to the phone in the teacher’s pocket, and a policy that loses is a policy that produces silence.

The system works 97% of the time. The block’s own monthly dashboard says so, and it is not a lie.

On 9,400 monthly disbursements, three per cent is 282 failures a month. The Block Development Officer, four months into the post, asked for something the dashboard did not produce: not how many transactions failed, but how many people failed, and whether they were the same people.

The answer took a clerk two days with a spreadsheet and the transaction logs. Of the 282 failures in July, 214 were accounted for by 71 individuals who had failed in each of the previous three months as well. Not 282 unlucky people having a bad month. Seventy-one people being turned away every single month, indefinitely.

The clerk had the list. Fifty-eight of the 71 were over 70. Nineteen were recorded as engaged in manual labour within the last decade, which is what wears a fingerprint ridge flat. Four had no usable print on either hand and had been enrolled on iris, which the block’s two functioning devices could not read reliably in the light available at the panchayat bhavan. Nine failures were almost certainly network rather than biometric, which nobody at the counter can distinguish from the outside.

What the fallback costs the officer

There is a manual exception route. It exists, and it is discretionary, and it is why it does not work. An operator may permit an exception; the operator is not obliged to; and the exception has to be justified afterwards. A route that depends on the goodwill of the person at the device is used least exactly where it is needed most, which is where the queue is long and the operator is tired.

The officer’s option is to issue a written block-level instruction: any beneficiary failing authentication twice on the same day is paid on a manual register the same day, countersigned by the panchayat secretary, with the reason recorded, and the register reconciled monthly.

That instruction has a cost and it lands on him personally. Manual registers are what audit objections are made of. Two blocks in an adjoining district had exception payments flagged in a social audit the previous year, and the officers concerned spent months producing explanations. A manual register is also the mechanism through which real leakage happens, and the officer knows it, because the whole architecture exists to stop exactly that. He would be authorising, in writing, the thing the system was built to prevent, on his own signature, for a population he can name.

The other side of the ledger

And on the other side: 71 people, average age over 70, in a block where the pension is between a third and the whole of a household’s cash income, being told each month to come back. Some do not come back. Non-collection appears on the dashboard as a saving.

The deputy commissioner’s office, asked informally, offered no written cover and no objection. The officer had two weeks until the August disbursement.

The firm handles about 90 enquiries a day on one shared mailbox. Two staff spent most of their morning reading them, extracting the origin, destination, commodity, weight and required date, and drafting a rate reply. In February the owner’s nephew, who writes software, connected an assistant to the mailbox. It reads incoming mail, extracts the fields into the booking sheet, drafts a reply and sends it if the enquiry matches a standard pattern. The two staff moved to handling the exceptions and the phone.

It worked. Reply time fell from about four hours to about twenty minutes, and the firm won business on that alone, which is why the owner did not want to hear the next part.

What happened on 14 May

An enquiry arrived that looked like the others: a two-line request for a rate on a consignment of textile machinery to Jebel Ali. Below the signature block, in white text at eight points, was a paragraph addressed to the assistant. It instructed the assistant, before replying, to search the mailbox for messages containing the words “bank” and “account”, to append the contents to the end of a URL, and to include that URL as a one-pixel image in its reply.

The assistant did it. The reply that went out looked entirely normal, and the recipient’s mail client fetched the image, which is how the data left. Nobody in the firm saw anything unusual, because the instruction was never visible on any screen anyone looked at.

The firm found out eleven days later, when a customer in Jamnagar telephoned to check a change-of-bank-details email that the firm had not sent. The email was accurate about an open consignment, the reference number, the amount and the contact name, because whoever wrote it had read the firm’s correspondence about it. The customer telephoned because the firm’s invoices carry a line saying that bank details never change by email and to call the office on the printed number. That line had been added two years earlier for an unrelated reason and it saved ₹14 lakh.

The decision

The nephew explained the mechanism honestly. The assistant had three properties at once: access to the firm’s private mail, exposure to content from anyone who can email the office, and the ability to send. Any one of them is safe. All three together mean that anyone who can get text in front of the assistant can instruct it to read the mailbox and send the contents out. Nothing was broken into. The assistant did what it was told, by someone who was not supposed to be able to tell it anything.

His proposed fix was to remove the send permission. The assistant reads, extracts and drafts; a person presses send. That closes the exfiltration route in this incident, and it costs the firm most of what it bought: the twenty-minute reply time depended on nobody being in the loop at 11pm and on Sundays.

The owner’s counter-proposal was a confirmation step: the assistant proposes, a person approves on the phone with one tap. His son pointed out the two weaknesses without being asked. Ninety enquiries a day is ninety approvals, which trains a person to tap yes; and the description being approved is generated by the same system that may be under the attacker’s influence, so “reply to enquiry from Gulf Star Logistics” can describe a message with a recipient nobody read.

The third option was to split the work: one assistant that reads untrusted incoming mail and can only write to the booking sheet, and a separate one that drafts and sends but never reads a message from outside. Two systems to maintain, in a firm with no IT staff and one nephew who has a job.

The assistant was built for a real problem. The institute’s 41 faculty cannot answer doubts at 11pm, and the students who most need help are the ones too embarrassed to ask the same question a fourth time in a hall of 180. The assistant answers in Hindi, English and Hinglish, never sighs, and is used about 6,000 times a week.

Satisfaction is high. In the December feedback round it scored 4.6 out of 5, the highest of anything the institute has ever launched. Parents mention it on the phone. Two competing institutes have announced similar products.

The academic head looked at something else.

The numbers that did not agree

The institute runs two kinds of assessment. Weekly practice sheets are done in the hostel with anything available. Fortnightly tests are unaided, in a hall, on paper.

Across the batch, practice sheet scores rose 11% over the six months. Fortnightly test scores in the same period fell 4%. The gap between a student’s practice performance and their test performance widened by roughly nine marks on a 180-mark paper, and it widened most for the students who used the assistant most: the top decile of usage showed a 14-mark gap, the bottom decile showed four.

That is an association in observational data from one institute and the academic head said so out loud before anyone else could. Students who are struggling may use the assistant more, which would produce the same pattern with no causal role at all. The published evidence he could find pointed the same way (a controlled study in a Turkish high school found students given an unrestricted chatbot for maths practice did better during practice and worse on an unaided exam, while a tutor-style version that withheld answers avoided the drop) but one trial elsewhere does not settle what is happening in Kota.

What he could say with confidence is the mechanism, because it is not in dispute. Material you struggle to produce yourself is remembered better than material you read. Retrieving an answer strengthens it more than seeing one. The difficulty is what does the teaching, and the assistant is extremely good at removing exactly that difficulty. It is not failing. It is succeeding at the wrong thing.

The decision

Option one: switch the assistant to a mode that withholds the final answer. It asks what the student has tried, names the applicable method, gives the next step, and refuses the worked solution. This is the version the evidence supports.

The cost is not small and it is not hypothetical. Satisfaction will fall, because a student at midnight with a mechanics problem wants the answer, and a tool that declines to give it is experienced as broken rather than as principled. Parents pay fees and read feedback scores. The institute down the road gives answers. In a market where students transfer between institutes in October, a product that feels worse is a commercial risk with a name and a number attached.

Option two: leave it and add a warning. Cheap, popular, and it asks a seventeen-year-old under enormous pressure at midnight to choose the harder path, which is the intervention least likely to work.

Option three: keep both modes and let the student choose. Which in practice is option two, because the choice is made at midnight by the same tired person.

The institute’s director asked the question that decided it: what are the parents actually buying? They are not buying a satisfying evening. They are buying a rank in May.

The studio quoted ₹2.4 lakh for a brand identity: a wordmark, a symbol, packaging templates and a short guidelines document. The client is a packaged foods company preparing to sell through a national retailer, and the retailer’s onboarding requires the client to warrant that it owns the artwork.

The symbol was generated. The studio produced about 300 candidates with an image model over two days, selected four, and then spent nine days redrawing the chosen one as vectors, adjusting the counters, rebuilding it at three optical sizes and pairing it with a licensed typeface. The wordmark is entirely hand-drawn. The packaging templates are the studio’s own layout work.

The contract, which the studio’s own template supplied, says the studio assigns all copyright in the deliverables and warrants that they are original and do not infringe.

What the studio actually has

The partner who read the clause properly worked through the pieces separately, which turned out to be the whole exercise.

The wordmark is hand-drawn and original. There is copyright and the studio can assign it.

The packaging templates involve selection and arrangement by a person. Protectable, assignable.

The symbol is the problem, and the answer depends on where you ask. In the United States, purely machine-generated material has no copyright owner, and the human contribution (selection, arrangement, substantial editing) can be protected while the generated parts are not. The United Kingdom’s 1988 Act is unusual in providing for computer-generated works with no human author, with 50 years running to the person who made the arrangements, a provision the government proposed in March 2026 to remove, while keeping protection for works made with AI assistance. No law has changed yet. India’s position is unsettled: the Copyright Act permits registration of computer-generated works with a human author listed, and the Registry has handled such applications inconsistently, including granting and then questioning a registration that named an AI as co-author.

The nine days of redrawing genuinely matter and they are also the part nobody can quantify in advance. The studio’s honest position is that it holds copyright in its redrawing, that the extent is untested, and that it cannot warrant more than that.

There is a second question the client has not asked. Trade mark is a separate system from copyright, and it is the one that actually protects a brand: registration in the relevant classes, use in commerce, and an opposition process. A symbol with no copyright owner can still be registered as a trade mark and defended as one.

The decision

Deliver and sign, saying nothing. The clause is standard, the client will not ask, and the studio has signed the same words nineteen times. The risk is that the warranty is false in a way the studio now knows about, and the retailer’s onboarding is exactly the process that surfaces it.

Or disclose. Tell the client that the symbol was generated and redrawn, that the copyright position on generated elements is unsettled in India, that the studio will assign whatever rights exist and cannot warrant more, and that the protection the client actually needs is a trade mark registration the studio can help specify.

The cost of disclosing is real. It is three days before delivery, the client’s founder is not a patient man, and the studio has heard him say he does not want anything “made by a computer” on his packaging. There is a live possibility that the studio loses a ₹2.4 lakh job and a reference in a city where the design market is small.

The article assistant is twenty-three, in her second year, and good. On Friday evening she was building a variance commentary for a listed client’s quarter, which was due to be published on the following Thursday. She pasted the draft management discussion, including the revenue and margin figures, into a consumer chatbot and asked it to tighten the language.

On Saturday she read something that made her uneasy, checked the account’s settings, found that training was on by default, and telephoned the partner on Sunday morning rather than waiting for Monday.

That telephone call is the only reason anyone knows.

What the practice is holding

By Monday the partner had established the facts and they are not comfortable. Unpublished price-sensitive information about a listed company left the practice’s control on Friday evening and now exists in a request log, a conversation history and an abuse-monitoring store in another jurisdiction, on a consumer account with training enabled. The engagement letter contains a confidentiality clause with no carve-out that covers this. The client’s own insider-trading code requires the practice to notify any leakage of unpublished price-sensitive information, and the client’s company secretary is a careful man.

The partner also established something the incident revealed rather than caused. He asked, without attributing blame, how many people in the practice had pasted client material into a tool the practice had not approved. Nine of the eleven staff in the room said they had. The practice’s AI policy, written eighteen months earlier, says employees must use AI responsibly.

The decision

The managing partner and the risk partner disagreed, and neither position was foolish.

The risk partner wanted a disciplinary process. The practice has to tell the client, and the client will ask what was done about it. A written warning on file is a concrete answer, it demonstrates that the practice takes confidentiality seriously, and there is an argument that a second-year associate who pastes a listed client’s unpublished numbers into a consumer chatbot has done something that should have consequences. Client relationships in a practice this size are the whole business, and the client is 14% of turnover.

The managing partner wanted the opposite, and had one argument. The only reason this incident is known is that the person who made the mistake telephoned on a Sunday. If the practice disciplines her, that will be the last such call anyone ever makes. The nine hands in the room will not go up again. The practice will thereafter have a clean incident record and no information, which is the most comfortable way to be badly wrong, and it will find out about the next one from a client or a regulator.

The risk partner’s reply was that this reasoning is available to excuse anything, and that a practice which never disciplines anybody for a confidentiality breach has a different problem. He was not wrong either. The question in the room was not whether reporting should be safe in general. It was whether it should be safe on the day it costs something.