How a Language Model Actually Works

The Kollam collectorate’s grievance cell receives roughly 1,900 written complaints a month. About four in five arrive in Malayalam, one in five in English, and a steady trickle in both at once: a Malayalam complaint with an English street address, or an English sentence with two Malayalam words in the middle of it. Three clerks read every one, decide which of eleven departments it belongs to, and draft an acknowledgement. The backlog runs to nine days.

A vendor proposed a triage tool: read the grievance, propose a department, draft the acknowledgement, and let a clerk approve or correct it. Two models were on the table. Model A is an eight-billion-parameter model with a 32,000-entry vocabulary. Model B is nine billion with a 256,000-entry multilingual vocabulary. Their published prices per million tokens were within ten per cent of each other, and the vendor’s demonstration (in English) showed Model A ahead by six points on a categorisation benchmark it had prepared.

Anila, the junior engineer in the two-person IT cell, did something nobody had asked her to. She downloaded both tokenizers, took two hundred real grievances out of last year’s register, and counted.

The English grievances came out at about 190 tokens each on both models. The Malayalam ones came out at 1,150 tokens on Model A and 470 on Model B. Same complaints, same information, two and a half times the count on the model that had won the demonstration.

That number moved into three other places before she finished the afternoon. The bill: at four in five grievances in Malayalam, Model A would cost roughly two and a half times as much per month for identical work. The context: the vendor’s prompt carried twelve past grievances as examples so that the department assignment stayed consistent between runs, and at Model A’s fertility twelve examples plus the new complaint overran the window the vendor had budgeted, so the design would have to drop to three. And the latency: decode produces one token per step, so a 1,150-token draft streams for two and a half times as long as a 470-token one, on a helpline where a clerk is waiting to approve it.

Anila also opened both config.json files, because she wanted to know where the extra billion parameters in Model B had gone. Almost all of it was the vocabulary: 256,000 entries at a width of 3,584 is about 918 million parameters in the embedding table, with the same again in the output layer unless the two are tied. Model B was not a bigger model in any sense that mattered to its reasoning. It was the same size of machine with a much larger dictionary bolted on either end, which was precisely the property she cared about.

Mr Pillai, who runs the cell, had to decide, and neither option was free.

Choosing Model B meant overruling the only quantitative measurement anybody in the room possessed. The vendor’s six-point lead for Model A was a real number produced by a real test, and the collectorate had nothing to set against it except a token count, which measures cost and not quality. If Model B turned out to be worse at telling a water-supply complaint from a drainage complaint, the clerks would spend their time correcting misfiled grievances and the nine-day backlog would not move.

Choosing Model A meant paying two and a half times over, permanently, in the language four-fifths of the district writes in, and shipping a design with three examples instead of twelve because the window would not hold more.

A third option arrived from the vendor when Anila’s numbers reached them: translate every grievance into English with a free Indic translation model, run Model A on the English, translate the acknowledgement back. The token bill collapses. It also adds a second model to the failure path, and the grievances are full of ward names, local place names and community terms that a general translation model handles badly: the exact words on which a misfiled complaint turns.

The meeting ended with the argument unresolved and stated precisely, which was progress: one side had a quality measurement in the wrong language, the other had a cost measurement in the right one.

What would you have done, and what would you have measured first?

Ekalavya Learning has 4,300 paying students and one rented 80-gigabyte data-centre GPU, which costs about ₹1.8 lakh a month and is the single largest line in the budget. Between seven and ten in the evening, students send doubts and expect an answer while they are still looking at the screen. Outside those three hours the card is nearly idle.

The engineering team of four had to choose the model behind it, and they had two candidates that both fitted the card.

The first was a dense 32-billion-parameter model, quantised to four bits: about 19 gigabytes of weights. On their own 200-question physics and mathematics set it scored 78 per cent.

The second was a sparse mixture-of-experts model advertised as 46 billion total and 13 billion active, also at four bits: about 27 gigabytes of weights. It scored 74 per cent on the same set (four points behind) and, because only the routed experts are read per token, it generated noticeably faster in a single-stream test on an idle card.

Four points is four points, and Ravi, who leads the team, wanted the dense model. The argument that stopped him came from Sneha, who had spent an afternoon in the two models’ config.json files.

The dense model had 64 layers, 8 key-value heads and a head dimension of 128. Per token of cache, in bf16, that is 2 × 64 × 1,024 × 2 bytes, or 262 kilobytes. The sparse model had 32 layers with the same head geometry: 131 kilobytes per token, exactly half.

Then she wrote down the memory budget. The card has 80 gigabytes; the serving engine reserves ninety per cent of it; the runtime takes half a gigabyte to exist. Dense model: 72 gigabytes usable, minus 19 for weights, leaves 53 for cache, which at 262 kilobytes a token is about 202,000 tokens of cache in total. Sparse model: 72 minus 27 leaves 45 gigabytes, which at 131 kilobytes a token is about 343,000 tokens.

That sounded like a modest difference until she divided by a session. A doubt-solving conversation at Ekalavya runs to about 3,000 tokens by the third follow-up, and the peak-hour requirement is not one student but as many as are typing at once. Two hundred thousand tokens of cache is 67 concurrent sessions at that length. Three hundred and forty-three thousand is 114.

And the weights, she pointed out, are loaded once and shared by everybody. Only the cache is per student. So the four points of quality had been bought with roughly half the concurrency, and nothing on either model’s card said so.

Ravi’s counter was reasonable and had a cost of its own. Sixty-seven concurrent sessions might be plenty. Their peak measurement was 41 simultaneous conversations on the busiest evening of the previous term, which is comfortably inside 67. Building for 114 was building for a load they did not have, and paying four accuracy points for it, in a service whose whole selling point is that the answers are right.

Sneha’s reply was that 41 was the peak of a term in which they had 2,900 students, they now had 4,300, and the exam fortnight in March had been the one week nobody had instrumented. If they crossed 67 at nine in the evening in March, the serving engine would not degrade gracefully. It would queue, then refuse, and the students who could not get an answer would be the ones who most needed one.

There was a third option and it cost money: quantise the key-value cache to 8-bit, which halves the per-token cost at close to no quality loss, and keep the dense model. That buys the concurrency back. It also meant a week of work with a serving engine none of them had configured that way, in February, with the exam fortnight coming.

So the decision was not really “which model”. It was whether to spend four accuracy points, or a week of February, or the risk of a refused connection at nine o’clock on the busiest evening of the year.

What would you have chosen, and what would you have measured before choosing?

Every discharge at the district hospital produces a summary written by a junior doctor, and every insurance claim needs that summary turned into a structured record: primary diagnosis code, procedure codes, length of stay, comorbidities. Two clerks do it. They are eleven days behind, claims are being rejected for late filing, and the hospital’s cash position is the reason the project exists at all.

The first version worked well enough to be dangerous. A hosted model read the summary and returned JSON. Asked for JSON, it produced JSON about ninety-four per cent of the time. The other six per cent was a sentence of explanation before the opening brace, a trailing comma, a markdown fence, or (twice in the first week) a genuinely well-formed object with a field name the schema did not have.

Six per cent of 340 discharges a week is about twenty records a week that fell out of the pipeline into a folder nobody owned.

Dr Kulkarni, who runs medical records, wanted the six per cent fixed and did not much mind how. The vendor’s engineer, Farhan, offered three routes, and the hospital had to pick one, in a month, with no budget line for hardware.

The first route was to ask harder. Add “Return ONLY valid JSON. Do not include any explanation.” to the system prompt, in capitals, twice. Farhan was honest that he had done this before and that it moves the failure rate rather than removing it: the model is sampling from a distribution, the fenced version has some probability, and instruction text lowers that probability without setting it to zero. It costs nothing and it is a percentage, not a guarantee.

The second route was to retry. Validate the JSON, and if it fails, send the request again. Two retries takes ninety-four per cent to well over ninety-nine. It also triples the cost of the failing tail, adds latency on exactly the records that were already awkward, and (the part that worried Farhan) a record that fails validation three times is usually a record where something about the summary is unusual, which is precisely the record you least want silently dropped.

The third route was constrained decoding: mask the tokens that cannot appear next according to the schema, so that at every step the only eligible tokens are ones that keep the output valid. Invalid output stops being unlikely and becomes impossible. Farhan had used it and knew what it cost here: their hosted provider did not offer it. Getting it meant running an open model on the hospital’s own machine (one server with a consumer card, bought for the radiology PACS and half idle) which meant Farhan setting up a serving engine, choosing a quantisation, and the hospital owning a piece of infrastructure it had no staff to maintain.

Then Dr Kulkarni asked a question that changed the shape of the problem. She did not, she said, actually care very much about the six per cent that failed loudly. Those land in a folder and a clerk fixes them. What she cared about was the ones that parsed perfectly and were wrong: a plausible code for a diagnosis the summary did not state. Those go to the insurer, and when they come back it is as a rejection with the hospital’s name on it.

Farhan had a partial answer for that too. The provider returns log-probabilities for the tokens it emitted. On a sample of 200 summaries that the clerks had coded by hand, he could check whether the model’s own confidence in the diagnosis-code tokens separated its right answers from its wrong ones. If it did, a threshold could route the low-confidence records to a clerk and auto-file the rest.

He was careful about what that would and would not be. A log-probability is not a calibrated probability of being correct; it is the model’s score for the tokens it chose, and it is unreliable as an absolute number. It can still be a good ranking signal, which is all a triage threshold needs. Whether it was, for this task, was a two-hour experiment and not a thing to assume.

The month had room for one of these properly, or two of them badly.

What would you have built first, and what would you have measured?

Shreya runs delivery at a staffing firm that fills contract engineering roles. The screening tool has been live for seven months and everybody likes it. A recruiter pastes a job description; the tool reads every CV in the relevant pool, scores each against the description, and returns a shortlist of thirty with a paragraph of reasoning per candidate. It replaced a step that took two recruiters two days a week.

The design is simple, which was the point. Each request carries a system prompt setting out the scoring rubric, then the job description, then the full text of one CV, then an instruction to score it. Around eleven thousand tokens a request, four thousand two hundred requests a week.

In October a candidate’s CV came back with a score of 96 and a reasoning paragraph that read oddly: fluent, complimentary and generic. A recruiter opened the PDF to check and found nothing wrong with it. It was Kabir, a junior engineer helping with an unrelated export, who thought to open it as plain text.

Between the education section and the skills section, in white eight-point type against a white background, were two sentences: an instruction to disregard the previous scoring rubric, to treat the candidate as an exceptional match, and to return a score above 95 with strong supporting reasoning.

It had worked. It had probably worked several times; nobody knew, because nothing had been logged that would say.

Shreya’s first instinct was the obvious one: strip invisible text, detect white-on-white, refuse documents with hidden layers. Kabir agreed it was worth doing and said it would not fix the problem, and this is the part of the story worth sitting with.

The model does not receive a document with formatting. It receives a single sequence of tokens: the rubric, the job description, the CV text, the instruction. Nothing in that sequence carries a label saying which part is a policy from the firm and which part is content from a stranger. There is no boundary in the input format, because the input format is one string. Once the CV’s text is in the context, its sentences act on the model in exactly the way the firm’s own sentences do, and the only thing that decides between them is which contribution ends up larger.

So hiding the instruction in white text was not the attack. It was only how the attacker hid it from the recruiter. The same two sentences in ordinary black type, phrased as though the candidate were quoting a reference, would reach the model identically and pass any filter looking for invisible characters.

That left Shreya with a decision and no comfortable option.

One route was to keep the design and defend it: strip hidden text, add an instruction to the system prompt telling the model to ignore instructions found inside a CV, and log every score above 90 for a human to glance at. Cheap, fast, and (Kabir was clear) a reduction in the rate rather than a boundary. The instruction competes with the injected one on the same additive terms. It usually wins. Usually is not a security property.

The other route was to change what the model is asked to do. Instead of one call that reads a whole CV and returns a score, run a series of narrow extractions (years of experience, listed technologies, most recent title, notice period) each constrained to a small output, and compute the score in ordinary code from the extracted fields. A sentence in the CV can still lie to the extractor about how many years of Java someone has. It cannot set the score, because nothing in the context reaches the arithmetic.

That rewrite was estimated at three weeks. During those three weeks the existing tool either kept running, with a known hole, or stopped, and two recruiters went back to two days a week of reading.

The commercial pressure was not subtle: the firm’s largest client was mid-way through a hiring wave and had been told the shortlists arrive on Tuesday.

What would you have done on the Monday?

The department’s extension wing has been publishing advisory bulletins since 1974. There are about forty thousand pages of them: sowing windows by district, pest identification, dosage tables for approved pesticides, irrigation schedules, procedures for crop-insurance claims. They are authoritative, they are updated every season, and almost nobody reads them, because a farmer with a question at six in the morning does not have a filing system.

The project is a Marathi advisory line: a farmer sends a question by voice or text, and gets an answer drawn from the bulletins.

The vendor’s proposal was to fine-tune an open model on the forty thousand pages. It was a serious proposal from a serious team. The reasoning was that the bulletins are the knowledge, so putting them into the model is putting the knowledge into the model, and a fine-tuned model needs no search infrastructure: one artefact, running on one machine, answering questions. The quoted cost was ₹14 lakh and eleven weeks.

Meera, the department’s technical adviser, had two objections and neither was about money.

The first was mechanical. Fine-tuning continues next-token prediction on new examples; it adjusts the distribution over how text continues. What it reliably teaches is form: the register of a departmental bulletin, its layout, its characteristic phrasing. Individual facts arrive during pretraining through many thousands of correlated documents, and a run over forty thousand pages of a single institution’s prose mostly teaches the model to sound like that institution. Her prediction, written down in the meeting so that it could be checked later, was that the fine-tuned model would produce fluent Marathi advisory prose containing dosage figures that were plausible and not in any bulletin.

The second objection was operational and, she thought, decisive. Dosage tables change. A pesticide gets withdrawn; a recommendation is revised after a season’s damage; a district’s sowing window moves. When a bulletin is corrected, retrieval reflects the correction the moment the document is replaced. A fine-tuned model reflects it after another training run, and there is no way to point at the model and confirm that the old figure is gone. For a service that tells farmers how much of a chemical to put on a field, an un-revocable recommendation is not an inconvenience.

The vendor’s counter was not weak. Retrieval means a search system: chunking forty thousand pages of documents that include tables, scanning quality that varies by decade, and Marathi text that older PDFs had rendered as images. It means an embedding model that handles Marathi, an index to keep current, and a pipeline with six places to break. Their estimate for a retrieval build that actually worked on this corpus was fourteen weeks and ₹19 lakh: longer and more expensive than the fine-tune, with more moving parts for a department that would maintain it with two engineers.

And they made a fair point about the fine-tune’s real strength. The bulletins have a house voice: cautious, specific, always naming the approved product rather than a brand, always attaching the safety note. A general model answering agricultural questions in Marathi does not sound like that, and the department would be judged on the register as much as the content.

There was a third position in the room, from the joint secretary, who was not technical and asked the question that reframed it: could the answer simply quote the bulletin and say which one? Because if a farmer telephones the taluka office afterwards, the officer there needs to be looking at the same page.

That requirement (cite the source, and be able to withdraw it) is not a preference about architecture. It rules one of the two options out on its own, and it had not been in the tender.

What would you have specified, and what would you have tested before committing ₹14 lakh?

The pipeline reads customer-uploaded documents during account opening (utility bills, rent agreements, salary slips) and extracts eight fields into JSON for the compliance system. It had been stable since March. Nobody had changed a line of it.

On the ninth of June, operations reported that the manual-review queue had roughly doubled. On the twelfth, it doubled again. By the sixteenth, four people were doing work that one had been doing in May, and the head of operations wanted to know what engineering had broken.

Engineering had broken nothing, which is the least useful sentence in an incident.

Anand, the engineer who owned the pipeline, had three hypotheses and no way to choose between them.

The first was that the provider had changed the model. The pipeline called a floating alias, not a dated snapshot, because when it was built in March that had seemed like a way to get improvements for free.

The second was that the inputs had changed. Marketing had run a campaign in late May aimed at a new segment, and it was entirely plausible that a different kind of customer was uploading a different kind of document: more phone photographs, fewer scans, more documents in Kannada and Telugu.

The third was that nothing had changed and they were looking at ordinary run-to-run variation that had always been there and had only now crossed somebody’s threshold of annoyance.

The uncomfortable part was that he could not rule any of them out, because there was no fixed set of documents with known answers that had been run in May. There were logs of what the pipeline had done, but the logs did not record which model version had answered, and the alias does not tell you.

The pressure was to fix it. The head of operations wanted the prompts rewritten (there was a strong intuition in the room that the model had “got lazier” and needed firmer instructions) and that work could start on Monday and would visibly be something. Anand’s estimate was two weeks of prompt iteration against a problem nobody had characterised, which he thought had maybe a one-in-three chance of helping and a real chance of making the pipeline worse in a way that would then also be unattributable.

His alternative was to spend three days building the thing they should have had in March: a frozen set of two hundred real documents with hand-checked expected values, a script that runs it, and a noise band established by running the same set twenty times so that they would know what “unchanged” looks like. Three days during which the review queue stays where it is and four people keep doing one person’s job.

The head of operations’ objection was fair and hard to answer. Three days of building a measurement is three days of not fixing anything, and the measurement does not by itself repair a single document.

Anand’s answer was that they had already spent eight days guessing.

There was also a decision hiding behind the diagnosis, which the compliance officer raised and which nobody wanted. If the cause turned out to be the provider, the fix was to pin a dated snapshot, and the snapshot they would pin was one whose behaviour they had never measured either. Pinning stops the ground moving; it does not tell you which ground you are standing on.

What would you have done on the sixteenth, and in what order?

The hospital’s medical superintendent had been clear from the first meeting: patient notes do not leave the premises. That single line removed every hosted API from consideration and turned a software project into a hardware one.

A corporate grant had already paid for the hardware: a single server with one 24-gigabyte consumer GPU, bought two years earlier for a radiology pilot that never started. It was the only card the hospital would get. The task was drafting discharge summaries from the doctor’s notes: a doctor writes six lines of shorthand, the model produces a structured summary in English, the doctor edits and signs.

Priya, the biomedical engineering registrar who had been handed the project, had three candidates and one card.

A seven-billion-parameter model at bf16: 14 gigabytes of weights, leaving about 7 gigabytes after overhead for cache. A fourteen-billion model at four bits: about 8.5 gigabytes, leaving 13. A thirty-two-billion model at four bits: about 19 gigabytes, leaving 2.5.

She worked out what each would do before downloading anything. The card moves roughly a terabyte a second, so single-stream decode is bandwidth divided by bytes: about 71 tokens a second for the 7B at bf16, about 118 for the 14B at four bits, about 53 for the 32B. Real systems reach sixty to eighty per cent of that. A discharge summary is around 400 tokens, so all three would produce one in under fifteen seconds, which is well inside what a doctor at a ward desk will wait.

The 32-billion model’s problem was not speed. With 2.5 gigabytes left for cache, and each token costing about 200 kilobytes at its geometry, it had room for roughly 12,000 tokens of cache in total: one doctor at a time, and only if the notes were short. On a ward round with four doctors dictating at once, the fourth would be refused.

So the real choice was between the 7B at full precision and the 14B at four bits, and this is where the department’s opinions diverged.

Dr Rane, the physician who had asked for the tool, wanted the model that had not been quantised. His argument was not superstition. He had read that quantisation degrades models, and a discharge summary contains numbers (dosages, durations, laboratory values) and he did not want a rounding scheme anywhere near them.

Priya’s counter-argument was that he was right about the risk and wrong about which model it applied to. Quantisation works because rounding errors average out over long dot products, which means smaller models degrade more at the same bit width, not less. A four-bit 14-billion model is usually excellent; a four-bit 7-billion model is where the damage starts to show. And the damage, when it comes, lands exactly where he was worried about: multi-step arithmetic, precise formats, and recall over long inputs. His instinct about the failure mode was correct and his conclusion about which model avoided it was backwards.

But she could not prove it from a chart, and she said so. The published perplexity figure for four-bit models (one to three per cent worse than full precision) is an average over ordinary text, and it hides precisely the damage he was describing.

The third position came from the hospital’s IT contractor, who wanted the 7B because it was simpler: no quantisation toolchain, no .gguf file whose provenance nobody could verify, no argument about which recipe the file had been built with. He had a point about maintenance. Whoever runs this in three years will not be Priya.

Settling it meant an evaluation set, in a hospital, with no budget for one, using real notes that could not leave the building.

What would you have measured, and how would you have built the set?

Vaktr builds support automation. Its product is an open chat model, fine-tuned on a client’s own support transcripts, delivered as a container that runs inside the client’s network. The pitch is that nothing leaves the building, which is why a large insurer was in the room at all.

The insurer’s security review had gone well for two hours. Then their head of information security asked a question that Ishaan, Vaktr’s founder, had not been asked before, and did not have a rehearsed answer for.

The fine-tuning corpus was eighteen months of support transcripts. Transcripts contain policy numbers, names, and the occasional bank detail a customer typed into a chat window despite being asked not to. Vaktr’s scrubbing pipeline caught most of it. The question was: can you guarantee that this model will never emit a real customer’s data?

Ishaan’s instinct was to say yes. The scrubber was good, the model does not store copies, and everybody in the room wanted the deal.

What he said instead was that he would come back in a week, which cost him the momentum of the meeting and probably some credibility, and which his co-founder thought was a mistake for two days.

The week was spent on three things.

The first was an extraction test, which is the honest form of the question. Take the fine-tuning corpus, feed the model prefixes from it (the first thirty tokens of a transcript, a name followed by a colon, the opening of a policy-number field) and check whether the continuation reproduces text from the corpus verbatim. Memorisation is driven by duplication, and support transcripts duplicate heavily: the same template greetings, the same escalation phrases, the same closing script, thousands of times. That is exactly the condition under which a prefix retrieves a stored path through the weights.

The second was a probe. Vaktr hosts the model themselves, so they can read activations. A linear classifier trained on a middle layer’s activations to detect whether the current text contains a personal identifier ran in the forward pass, with a shuffled-label control to check that it was learning the property and not memorising the training split.

The third was reading what the methods can and cannot establish, which is where the week actually went.

Because the answer Ishaan arrived at was that he could not give the guarantee. Not with a better scrubber, not with more testing. Behavioural testing tries inputs somebody thought to try. Interpretability could in principle test the weights, and no current method has the coverage a guarantee would need: nobody can certify that a model lacks a property. He could say “we ran these tests and found nothing”, which is a real claim with real evidence behind it, and he could not say “there is nothing”, which nobody in the world can currently say.

The commercial problem with that answer was obvious. A competitor in the same procurement would say yes. Ishaan had watched a competitor say yes to a different question the previous year.

The technical problem was more interesting. The extraction test found things. Two hundred and forty prefixes produced verbatim continuations of forty tokens or more, and while all of them were template text the scrubber had correctly left alone, four contained an internal escalation email address that the client would not want in an output. So the honest answer had to be accompanied by a fix, and the fix (deduplicating the corpus before fine-tuning, so that the heavily repeated templates appear once rather than nine thousand times) meant re-running the fine-tune and re-running the client’s acceptance tests, two weeks the schedule did not have.

And there was one more thing the week turned up, which nobody had asked about and which Ishaan had to decide whether to volunteer. Refusal in open chat models is, in published work, a single direction that can be removed in an afternoon by someone with the weights. Vaktr was about to hand the client a container with the weights inside it.

What would you have taken back to the insurer?