Choosing and Using the Tools

Sahyog Vidya Trust awards forty scholarships a year and receives about nine hundred applications for them. Each arrives as an email with a short statement of circumstances. A volunteer, Deepak, a second-year engineering student, built a sorter for it last March: paste the statement, and a free chat assistant returns one of six categories: first-generation learner, single-earner household, disability in the family, migrant household, girl child in a low-enrolment block, or none of these. The category decides which reviewer reads it and, for two of the six, which pot of money it is considered against.

It ran for seven months. Once a fortnight a trustee, Mrs Fernandes, read a sample of forty sorted applications against the trust’s own written rules, and agreement had sat between ninety and ninety-five per cent all year. Nobody had touched the prompt since May.

On a Tuesday in March her sample came back at sixty-eight per cent. The errors were not scattered. Applications mentioning a father who had died were landing in single-earner household, when the trust’s rules put a bereaved family into a separate review with a different reviewer and a different maximum award.

Shobha, who runs the programme, wanted to move to a different product that morning. Eleven days remained before the deadline and four hundred applications were still to arrive. Her argument was that the sample was the only quality check the trust had, that it had failed, and that spending the day investigating a free tool was a day the applications did not stop coming.

Deepak’s argument was that he did not know what had changed, and that a different product would be a different unknown. He had not edited the prompt. The trust was on a free account whose model picker had said Auto since the day he created it (a routing label, not a model name) and he had never looked at it in seven months. He could name three things that might have moved and had evidence for none of them.

The first was the model. Free tiers move you to a smaller model once an allowance is used, and the trust’s sorting runs in bursts, so the afternoon batch might be answered by something different from the morning batch. The second was the product’s own instructions, which the company rewrites without telling anyone. The third was the applications themselves. A new district scheme had opened in Gadchiroli in February, and those statements read differently: longer, and frequently describing two hardships at once.

Deepak’s prompt held six rules in a single paragraph.

Shobha’s counter was fair. Whatever had moved, the trust still had to sort four hundred applications correctly, and diagnosis is not sorting. She offered him until four o’clock, and pointed out that she could put two volunteers on manual sorting for the rest of the week, which would cost the trust about nine hours of work it had not budgeted for and would at least be work whose failures somebody could see.

Deepak’s objection to that was that manual sorting had its own error rate and nobody had ever measured it. The ninety-three per cent figure was agreement between the tool and Mrs Fernandes, not between the tool and the truth. He did not know how often two volunteers would agree with each other on the same forty applications, and neither did she.

Mrs Fernandes, when asked, made the point that decided the afternoon. She had been sampling for seven months and she had nothing written down except a tally. She could not say whether the tool had ever been good at bereavement cases, because bereavement cases are perhaps one application in twenty and her samples were forty at a time. It was entirely possible that the category had always been wrong and March was simply the month a sample had caught it.

That changed the shape of the problem. If the fault was seven months old, then a hundred and some applications had been misrouted since September, several of them to a smaller pot of money, and that was a separate matter from the deadline.

The thing all three agreed on was that the question was not which tool to use. It was which of the three layers had moved, or whether anything had moved at all, because the answer determined whether the fix was an hour or a fortnight, and because the trust had nothing written down that could tell them.

What would you have done with the afternoon, and what would you have wanted to exist before that Tuesday?

Advocate Anjali Kulkarni takes consumer disputes, tenancy matters and small commercial work. Her associate, Rehan, is eighteen months qualified. Most of the drafting is in English; the client letters and the statutory notices are in Marathi, and the register of a legal notice in Marathi is not something either of them will accept an approximation of.

One of the two products they were considering had an annual plan at eighteen per cent off, and the offer ended on Friday. That is what made this a decision rather than a discussion.

Anjali had read a comparison that gave Tool B a four-point lead over Tool A on a reasoning benchmark and two points on a coding one. She had also used Tool A on and off for eight months and liked it. Rehan pointed out two things about the comparison. Neither of them writes code. And neither benchmark contained a single item in Marathi, which is the part of the work where a bad answer is unusable rather than merely irritating.

His proposal was a blind test: twelve real prompts from their own files, run through three candidates in one sitting, answers labelled A, B and C, scored against criteria written before any answers were read, and the mapping turned face down until the end. He wanted Thursday afternoon for it.

Thursday had two hearings. That was the cost on Rehan’s side, and it was not small in a two-person practice in a week with a filing deadline.

The cost on Anjali’s side was less visible and larger. An annual commitment in a field where the right tool changes within six months converts a reversible choice into an irreversible one, and the saving was about fourteen hundred rupees: less than an hour of her time. And a choice made on a benchmark that contained none of her work would be re-litigated every time somebody published another one.

There was a third problem that Rehan raised and Anjali initially dismissed. Her eight months of favourable impressions of Tool A came from an account that had been told, across hundreds of conversations, that she practises tenancy law in Maharashtra. A tool that has learned your domain looks better on your domain. Comparing that account against two fresh ones is not a comparison of tools.

They argued about the number of prompts. Anjali thought six was enough and that she would know which tool was which regardless of the labels. Rehan’s answer was that knowing which is which is exactly the thing the labels are for, and that if she could genuinely identify them, the blinding costs nothing and the score is still the score.

One prompt in the twelve was of a kind Anjali had never used deliberately: a question about a provision of consumer protection law whose answer she already knew, written out in advance, with the section number and the source and the date she last checked it. Rehan had wanted two of these. Anjali thought one was enough, on the reasonable ground that they were buying a drafting tool and not an encyclopaedia.

His reply was the argument that changed her mind about the shape of the test. They were not buying a drafting tool. They were buying something a junior would use on a Friday evening when nobody senior was in the office, and the failure that costs a practice money is not a clumsy sentence. It is a confident paragraph containing a provision that does not exist, in a letter that goes out over her name.

There was one more thing to settle before Wednesday, and it was the part they nearly got wrong. Rehan proposed scoring each answer out of two: usable as it stands, usable after a fix, or rewrite from scratch. Anjali asked what they were scoring against. Neither of them had written down what a good answer to the Marathi notice contained.

They spent twenty minutes on that before they ran anything: one line per prompt saying what a good answer would have in it. For the notice it was four things: correct statutory register, the demand stated once and plainly, a date for compliance, and no English word order dressed in Marathi. Written afterwards, those criteria would have rearranged themselves around whichever answer impressed them first.

What would you have run, and with the discount expiring, when would you have run it?

The department of computer applications has forty lab machines with eight gigabytes of memory each, a shared Windows image, and students in three batches. Nearly all the students also have phones: mid-range Android, storage close to full, data bought in bundles. The head of department, Mr Selvaraj, had forty-five thousand rupees for the semester and a project brief that required every student to use an assistant and to record what it cost them.

Four routes were put on the table at a staff meeting and the arithmetic disposed of one immediately. A hundred and eighty personal subscriptions at twenty dollars a month is not a semester budget, it is a department’s annual one.

The second was a single department key on an open-source front-end installed on the lab server, with every student reaching it from a browser. One account to set up, one bill, works on the lab machines and from the students’ phones on the college network. Mr Selvaraj wanted this and wanted it on Monday.

Vimala, the lab technician, objected on two grounds, and her objections were the substance of the meeting. A single shared key means there is no way to tell whose request cost what, so a runaway spend is a mystery as well as a bill. And a student loop with a retry on failure, running against two thousand items on a Thursday night, can empty the semester’s budget before anybody sees an email.

Her alternative was a router with a separate key per student and a spending limit on each. That answers both objections exactly. It also means creating and later revoking a hundred and eighty keys, in a semester where she was already running three other labs, and it means somebody handling the prepaid balance every few weeks.

The fourth route was to tell students to use free tiers on their own devices, which costs nothing and produces no comparable work, because the tools cap at different points, some of the apps are not listed for older Android versions, and several students did not have three hundred megabytes free to install anything.

Then a student asked whether he could simply install a browser extension that put an assistant beside every page. The department had no policy. The extension’s permission line said it could read and change all data on all websites, and the lab machines are used to log into the college’s internal portal, where attendance and marks live.

Vimala had a second objection to the shared key that nobody expected, and it was the better one. The project brief required each student to record what their work had cost. With one key and no attribution, the only number available is the department’s total, which teaches nothing. The students would be writing down a figure somebody else had calculated for them, which is the opposite of the exercise.

A lecturer, Mrs Anitha, raised the case nobody had planned for. Eleven of the hundred and eighty students are in the evening batch and come to the lab twice a week. Whatever route the department chose had to work from a phone on a metered connection at home, or those eleven would do the project by borrowing somebody’s laptop on a Saturday.

That put a constraint on the answer that the budget had not. Voice and image uploads are expensive on a bundle; text is not. A photograph is a megabyte or more and a whole afternoon of text conversation is a few. Whatever was chosen had to be usable as text on a poor connection, and anything that streamed long replies over a connection that dropped would lose the answer along with the data spent on it.

Mr Selvaraj’s position was that Monday mattered: a route that exists imperfectly on Monday teaches more than a perfect one that arrives in week four. Vimala’s position was that the first runaway loop would end the experiment for everybody, and that the person who would be asked to explain it to the principal was her.

What would you have run, and what would you have spent the budget’s first rupee on?

The archive is forty-one thousand articles from 1998 onwards, about sixty per cent in Malayalam and the rest in English. The new site needs three things from each: three tags from a fixed list of sixty, a forty-word standfirst, and a flag on anything naming a living person, which routes the article to a legal read before it is republished.

Thomas, the editor, wanted the whole archive run through the strongest model available. His reasoning was not vanity. The archive is the paper’s only asset that a competitor cannot buy, and a bad tag is a piece of the paper’s history that nobody will ever find again.

Fathima, who is the entire technical department, ran twenty real articles first and wrote down what they actually used: about eighteen hundred tokens in and two hundred and forty out. On the frontier model that priced the job at a little under five hundred dollars. On the small model in the same family it was thirty-one. Thomas looked at both numbers and said the difference was worth it.

Then Fathima checked her sample and found she had taken the twenty articles from the top of a list, which sorted by date. Nineteen were from the last two years and four were in Malayalam. The archive is sixty per cent Malayalam, and Malayalam in its own script takes roughly three times as many tokens as the same content in English.

She re-sampled, stratified by language and by length. The average came out at about three thousand four hundred tokens in and four hundred and twenty out, nearly double her first estimate. The frontier model now priced at about six hundred and seventy-six dollars before retries, about nine hundred with the third she adds for them: a little over seventy-five thousand rupees. The small model came to forty-two dollars all in.

So the disagreement got sharper rather than easier. Thomas wanted the expensive run live, where he could watch it and stop it. Fathima wanted a cascade (the small model on everything, the expensive one only on flagged cases) and she wanted the escalation pass sent through the batch route at half price, which meant results arriving within a day rather than immediately and coming back unordered.

The launch was in four weeks. A prompt fault discovered on day three of a batch run is three days.

Thomas made an argument that deserved more respect than Fathima initially gave it. Seventy-five thousand rupees is a fortnight of a junior sub-editor’s salary, and the paper had spent more than that on worse things. If the expensive model meant not having to think about the tagging again, it was cheap. What he objected to was not the saving but the number of moving parts: two models, an escalation rule, a batch window and a confidence word, each of which could be the thing that went wrong on a Thursday.

Fathima’s answer was that she was not proposing to guess which items the small model could handle. She wanted to measure it on a hundred articles, against tags a person had written by hand, and to design the escalation from what that comparison showed rather than from an opinion about model sizes. Without that measurement a cascade is a guess that quietly ships worse work, and she said so.

The chief sub-editor, who would have to write those hundred sets of tags, wanted to know what happened to his afternoon if the answer came back saying the small model was fine. Fathima’s reply was that the hundred tags were not wasted either way: they were the only record the paper would have of what correct looked like, and the same hundred would be the check the next time anybody touched the prompt.

And there was a category neither of them wanted to route by confidence. A missed living-person flag is not a worse tag. It is a legal read that never happened, on an article about to be republished on a public website with the paper’s name on it.

What would you have measured before spending anything, and where would you have refused to let the cheap model decide?

Nikita Shah practises alone and keeps twelve clients. The largest is a garment exporter whose engagement letter contains an ordinary clause: client records may not be disclosed to any third party without written consent. She had never thought of it as an AI clause, because it was drafted in 2019.

The work in front of her was four years of ledger narrations: about forty thousand lines, most of them abbreviated, inconsistent between the two people who had entered them, and full of names. She wanted a model to group them, flag the ones that did not match a head of account, and draft the management letter that came out of it.

Her machine is a three-year-old laptop with sixteen gigabytes of memory and no separate graphics card.

Three options were on the table. A business-tier subscription at about twenty-two hundred rupees a month buys a contract saying the content is not used for training, a data processing agreement and a retention setting. All of that is real and none of it answers the engagement letter, because a data processing agreement does not make a disclosure into something other than a disclosure. A hosted open model is cheaper and faster and is also a third party. And a machine that would run something larger locally (thirty-two gigabytes, ninety-five thousand rupees) is four years of the subscription in a year where she was also replacing a printer and had just started paying a part-time assistant.

So she installed Ollama and an eight-billion-parameter instruct model on the laptop she had. In chat it was excellent: drafting, rewriting, tidying her own notes, about fourteen tokens a second, entirely adequate.

Then she pasted a single year of narrations into it. Thirty-eight thousand tokens. The machine slowed to a crawl, the fans came on, and the first word of the answer took over a minute to appear.

She assumed the model was too big for the machine. It was not. The weights were about four and a half gigabytes and had been fine all week. What had changed was the working memory for the conversation, which grows with the length of the context and on a document of that size had come to rival the weights themselves, on top of the runtime and everything else the laptop was already holding.

The decision in front of her was whether to buy the machine now, or to split the work (local for anything carrying the client’s identity, hosted for everything else) and accept that the ledger job would run in small batches overnight over several days rather than in an afternoon.

The case for buying was not only speed. Her practice is her own, there is no IT department, and she disliked the position of having the confidentiality of her largest client depend on a setting she might change by accident. A machine that could hold a larger model comfortably would let her stop thinking about the boundary altogether, which is a real thing to want.

The case against was the part that is easy to leave out of the arithmetic. Ninety-five thousand rupees is four years of the subscription she was comparing it against, and the subscription’s model improves underneath her for nothing while the machine holds still. A colleague in Rajkot had bought a graphics card eighteen months earlier for exactly this reason and had since spent a lost weekend on a driver problem after an operating system upgrade, and another evening re-downloading every model he had because a file format had moved on.

There was a third option nobody had put on the table, which was to ask the client. Nikita had not considered it because she had framed the whole question as a technical one about where computation happens, when the clause in the engagement letter is about consent and consent is something a client can give. The exporter’s finance director is a person she speaks to most weeks.

What would you have tried before spending ninety-five thousand rupees, and what would you have put in the letter?

Class teachers at the school write four to six lines of comment for every pupil at the end of term, about forty pupils a class, nine classes in the middle school. It takes an evening a class and everybody dislikes it.

In September a teacher, Mr Barla, started pasting the term’s marks and his rough notes into a free assistant and asking for a polished comment. The results were better than what he had been writing, and they were better still when he included the pupil’s name, because the sentences read more naturally. By December most of the middle school’s comments were being drafted this way. Nobody had decided anything. There had been no meeting.

At the parents’ evening a mother who works as a software tester asked a straightforward question: which service was used, and what happens to what was typed into it.

Sister Mary Grace, the principal, did not know. The IT coordinator found out in ten minutes. A free consumer tier. A setting in the account, switched on by default, permitting the conversations to be used to improve the model. A retention statement that said content is kept as long as necessary, which is not a period anybody can plan around. And a paragraph, as almost every policy has, saying that samples and flagged conversations may be read by people.

The vice-principal wanted the practice stopped that evening. The position is defensible and it is what a parent would expect to hear.

Mr Barla’s position was that the comments were better, that thirty-one teachers had just been handed back an evening a term, and that a ban would not return them to handwriting. It would move the practice to personal accounts on personal phones at home, where the school would know nothing about it at all.

Then a specific fact came out that changed the register of the conversation. One of Mr Barla’s comments had explained a pupil’s drop in attendance by referring to her father’s illness, because that was the honest explanation and he wanted the comment to be fair. He had typed it, with her name, into a free consumer account, with no policy in front of him and no ill intent whatsoever.

Three things followed from that one comment and none of them was about the assistant. The information was about a third person who was not the school’s pupil. It was health information, which in most data protection regimes sits in a more protected category than an address or a mark. And it had been typed into an account whose settings permitted the conversation to be used for training, by a teacher who had never been told there was a decision to make.

The IT coordinator raised a separate point that the meeting had not reached. Thirty-one teachers were doing this on whatever account they already had, which for most of them was the account they also use at home. Report comments, family circumstances, and the teacher’s own private conversations were accumulating in the same history, on devices that in several cases are shared with children and spouses. Memory features in some of these products carry context from one conversation into the next, which means a teacher who has spent a term discussing pupils may find that context appearing in a personal question.

The vice-principal’s answer to all of this was that it strengthened her case. The coordinator’s view was that it strengthened the opposite one, because every fact she had just listed was a fact the school had learned by asking, and a ban would end the asking rather than the practice.

The cost on one side was a rule that somebody had to write, a route that somebody had to find, and a check that thirty-one teachers were following it, in December, in the last fortnight of a term. The cost on the other was a ban that looked responsible to a parents’ meeting and would produce a worse, invisible version of the same practice by February.

What would you have written, and what would you have said to the parent?

The office holds twelve thousand four hundred trade licence applications from 2016 onwards, about fifteen thousand eight hundred pages. They are needed as searchable records with eight fields each: licence number, applicant name, trade category, ward, date of application, fee paid, date of issue and renewal status.

A vendor quoted fourteen rupees a page for what the proposal called AI-powered document intelligence. That is about two lakh twenty thousand rupees, six weeks, and a contract with somebody accountable at the end of it. The demonstration used three clean forms from 2023 and read all eight fields correctly from each.

A junior assistant, Rakesh Meena, asked one question in the meeting. Are these scans, or are some of them files the corporation’s own licensing software printed?

Nobody knew, so they checked. Four thousand nine hundred of the files had been generated by the office’s own system after 2021 and carried a text layer. A free command-line tool read every character of them exactly, in seconds, with no recognition involved and no possibility of a digit being misread. The remaining seven thousand five hundred were genuine scans of paper forms: many completed by hand, some skewed, a number of them photocopies of photocopies.

So the quotation covered four thousand nine hundred files for which the answer was already sitting in the file.

The decision that followed had a cost on both sides. The vendor’s price covered everything, arrived on a date, and came with somebody to complain to. Doing it in-house with free tools meant Rakesh and one colleague spending about five weeks of their own time in an office that is short-staffed, and if it went wrong there would be nobody to complain to but themselves.

Rakesh’s supervisor, Mrs Sharma, raised the point that settled the shape of the work. Two of the eight fields are numbers. A wrong trade category is an annoyance that somebody will notice and correct. A wrong fee in a licence record is a discrepancy against the treasury’s books that may not surface for years.

And nobody had tested the vendor’s tool on the worst material. Handwritten applicant names in Devanagari on a faint photocopy are a different problem from a clean printed form from 2023, and the demonstration had not contained one. When Mrs Sharma asked for a demonstration on twenty files the office would choose, the vendor’s representative agreed readily, which was itself informative, and then asked for a week.

Rakesh spent an evening of his own on the four thousand nine hundred text-layer files to find out whether the in-house route was fantasy. It was not. Exact extraction took minutes. The part that was not trivial was the fields: the forms changed layout twice between 2016 and 2025, the ward is written in three different formats, and about a fifth of the applications record the fee in words as well as figures, which disagree in a handful of cases.

That last discovery mattered more than the vendor’s price. It meant that whichever route the office took, somebody was going to have to decide what happens when a form says one thing in figures and another in words, and no tool was going to decide it for them.

Mrs Sharma’s own position was that the office had been offered a service and had not yet worked out what it was buying. Two lakh twenty thousand rupees for pages, or a database the treasury would accept? Those are different products and only one of them was in the quotation.

What would you have quoted for, and what would you have written into the contract?

Divya Menon keeps books and files returns for thirty-four small businesses. Over two years she has built six saved instructions that carry most of her repetitive work: an explainer she sends clients when their purchase register does not match what their suppliers filed, a bank-narration classifier, a payment reminder in Malayalam and English, a skeleton for a director’s report, a query list generated from a trial balance, and a plain-language note explaining a departmental notice.

They all run through an API from a free front-end on her desktop, against a pinned model snapshot rather than a moving name. She pinned it eighteen months ago, after a Tuesday morning when the classifier changed its mind about cash withdrawals and she lost a day finding out why.

The notice said the snapshot would be withdrawn in ninety days. It named a successor. The prices were different in both directions: input cheaper, output dearer, which for her work (long inputs, short outputs) probably meant slightly less.

Three weeks separated her from the start of filing season, which is six weeks of fourteen-hour days.

The argument for migrating immediately was that ninety days sounds long and contains the season. A regression found on the eighteenth of the month, with returns due, is not the same event as a regression found in June.

The argument against was that she did not have an evening. She was onboarding two clients that fortnight, and the snapshot would keep working throughout the season regardless. June is quiet. June is when you do this sort of thing.

There was a third item in the notice that she nearly skipped, because it was about the API and the API stores nothing of hers. But her front-end keeps its own conversation database on her own machine, along with the reference documents she has attached to it over two years, and she had never once tested whether its export produced anything usable.

She also noticed, reading the notice a second time, that it had arrived at the email address on the account. She is the only person in the practice, so the address is hers. It occurred to her that this is not true of most of her clients, several of whom have software accounts in the name of a relative who set them up and no longer reads that mailbox. A notice like this one, sent to such an address, is a notice nobody receives.

The pricing line deserved a second look too. Input cheaper and output dearer sounds neutral and is not, because it depends entirely on the shape of the work. Hers is long inputs and short outputs (a trial balance in, a query list out) so the change was probably slightly in her favour. A colleague who generates long client reports from short briefs would have read the same two numbers and found the opposite.

What she did not have was the thing the rest of this decision rested on. Her prompt file had eleven items and she had been meaning to add a twelfth since last year. Two of the eleven were prompts she had quietly rewritten after a tool failed them, which she now realised meant those two were no longer testing anything.

There was also a question she could not answer about her own setup. Six saved instructions lived in the front-end’s own database, and their only copy was in there. She had written them, refined them over two years, and never once pasted them into a file of her own.

What would you have done with the three weeks, and what would you have refused to do until June?