The Open Model Ecosystem

Ketki and her three colleagues had built the thing properly. Their app took a photograph of a maths problem from a Class 10 textbook, read it, and walked a student through the solution in Marathi or English. They had chosen a reasoning model because a wrong early step ruins a proof, and the distilled reasoning model they picked was good enough that two teachers at a pilot school had stopped correcting it.

The repository they had built on was DeepSeek-R1-Distill-Llama-8B. The DeepSeek papers were public, the R1 release was MIT, and MIT is about as free as a licence gets. Ketki had written “MIT” in the one-line technology section of the deck she showed schools, and nobody had asked a second question about it.

The second question arrived from a procurement officer at a chain of eleven schools in Nashik, who wanted a signed statement of every third-party component and its licence before a purchase order could be raised. It was a form. Her lawyer filled it in and then stopped at the model row and asked what the model was built on.

The answer was on the model’s own card. Its front matter said license: mit and named no base_model, but its License section said the model was derived from Llama 3.1 8B and originally licensed under Meta’s Llama 3.1 licence, and the base was Meta’s. DeepSeek-R1 is MIT; a distillation into Llama weights is not, because the weights being distilled into are Meta’s and Meta’s community licence travels with them. Ketki had been reading the licence of the model that produced the training traces rather than the licence of the model she was actually running.

What the real licence asked for

None of it was onerous in isolation. The Llama community terms wanted “Built with Llama” displayed prominently. They wanted a specific attribution line in a notice file. They required that any model she derived from it have a name beginning with “Llama”. And she had fine-tuned an adapter on four thousand solved problems and called the result Ganak. They attached an acceptable use policy that Meta could update after she shipped, and they stopped being free above seven hundred million monthly active users, which was not her problem and would be the first thing any acquirer’s diligence checked.

The app icon, the splash screen and two thousand printed flyers for the Nashik schools all said Ganak. The flyers had gone to press eleven days earlier.

The two ways out, both expensive

Comply. Rename the model (not the app, only the model, which the terms actually constrain) to something beginning with Llama, add the attribution, put the badge in the About screen, reprint nothing. Cost: about a day of work, a conversation with the procurement officer explaining that the licence in her deck had been wrong, and a permanent dependency on a document another company can edit. Her lawyer’s note was blunt: you are agreeing now to terms that can change later, and you cannot prove which version you accepted unless you keep a copy with a date.

Swap. Move to Qwen3-8B, which is Apache 2.0 with no naming rule, no use policy and no user cap. Cost: the prompt was tuned against the distilled model’s habit of thinking before answering and did not transfer. The adapter would have to be retrained against a different base. Their forty-example evaluation would have to be rerun. Ketki’s estimate was nine working days, which put them past the start of the school term, which was the only window that mattered that year.

The third option, which one of the four argued for at length, was to ship and deal with it later. The licence was not being enforced against anybody they knew of. The procurement form could say MIT and nobody would check.

What made it a decision rather than a choice

The argument that ended it was not about risk of enforcement. It was that they were about to sign a statement of components for a school, and the statement would be false. Ketki said she was not willing to put her name on it, and that settled the immediate question without settling the larger one, which was what they should be running a year from now.

Vaanam Knits ships cotton jersey to buyers in Germany and the United Kingdom, and takes almost every domestic instruction on WhatsApp. The messages arrive in three registers that its two-person software team had to treat as three different problems: Tamil in Tamil script from the dyeing and stitching units, Tamil written in Roman letters from the sales staff, and English from the buyers’ agents.

Suresh, who had built the company’s order system and taught himself the rest, wanted the messages tagged automatically into four buckets so that a complaint never sat unread behind sixty delivery updates. He did the right thing and built an evaluation set first: sixty real messages pulled from a month of chat exports, twenty in each register, each labelled by the sales manager.

Three candidates and one clear winner

He tested three. A general 8B instruct model under Apache 2.0, running on the office machine. A 32B rented by the token from an aggregator, which was the expensive option and the one he expected to win. And an Indic-tuned 7B built on an open base by a lab whose Tamil work he had seen praised.

The Indic model won, and not narrowly. On Tamil script it was eleven points ahead of the general 8B. On Roman-script Tamil (the register nobody benchmarks and the one half his traffic used) it was fifteen points ahead, because the general model kept reading Tanglish as misspelled English and tagging a complaint about shade variation as a sample request.

The tokenizer told the same story from a different direction. Suresh encoded the same thousand messages with each tokenizer and counted. The Indic model spent 2.6 times fewer tokens on the Tamil-script messages than the general 8B did. That is not a quality argument; it is a bill, a latency figure and an effective context length, all at once, and it was the number that convinced the managing director that this was engineering rather than enthusiasm.

The line in the front matter

The model card’s front matter said license: cc-by-nc-4.0.

Non-commercial. Vaanam Knits is a business, the tagging would run inside a commercial operation, and no reading of the phrase made that a research use. There was no ambiguity to exploit and no lawyer to ask for a better answer, because the answer was on the page.

What was actually on the table

Ask for a commercial licence. Suresh wrote to the lab. The realistic view was that a forty-person exporter in Tiruppur was not the correspondent a research lab replies to quickly, and he had no date by which he could promise an answer.

Ship the general 8B anyway. Eleven and fifteen points worse, 2.6 times the tokens, and (this was the part the sales manager cared about) the specific failure was that complaints in Tanglish got mis-tagged, which is the one bucket the whole project existed to protect.

Route. Two models: the Apache-licensed general model on the English traffic, where it was as good as anything, and a separate, permissively licensed Indic model on the Tamil and Tanglish traffic. Suresh had found one, from a different family and a different lab, under Apache 2.0. It was four points behind the non-commercial model on Tamil script and six behind on Tanglish: worse than the winner, much better than the general model, and legal.

Do nothing, and keep the sales manager reading messages. The status quo is always an option and it had a real cost, which was that two complaints in the previous quarter had reached the buyer before they reached Tiruppur.

The tempting reading of this case is that the licence ruined a good technical decision. The more useful reading is that the licence was a property of the model as much as its Tamil accuracy was, and Suresh found it in the fourth minute of a ten-minute check rather than in the fourth month of production.

The assistant had worked for six weeks. A student typed a physics doubt, the model answered in four or five lines, and a teacher reviewed anything flagged. It ran in a notebook on a rented GPU during development. On Thursday the institute’s system administrator, Farhan, moved it to the machine it would actually live on: a desktop with a 12 GB card under the server-room air conditioner, running a GGUF file through a local server so the pilot would not depend on the institute’s unreliable link.

On Thursday evening it stopped stopping.

A student asked why a block on an incline does not slide. The model answered correctly in four lines. Then it wrote another paragraph. Then it wrote Student: and invented a follow-up question, answered that, invented another, and continued until it hit the token limit five hundred tokens later. Every answer was like this. In the notebook, the same model, the same prompt, the same question had stopped cleanly.

What Farhan checked, in order

He had two days and he started with the cheapest checks.

Was a special token visible in the output? He turned off the server’s own formatting and looked at the raw text. There it was: a literal end-of-turn marker sitting in the middle of the reply, printed as characters rather than acted on. The model had said it was finished at exactly the right place. The runtime had not noticed.

Why had the runtime not noticed? He compared the two files. The original repository’s generation_config.json listed three end-token ids: an end-of-text token and two end-of-turn variants. The GGUF he had downloaded (a third-party conversion, five thousand downloads, no card beyond a line saying which quantisation it was) carried one. The other two had not survived the conversion.

Was anything else wrong with the file? This was the question he could not answer quickly. The same conversion that dropped two stop tokens writes the chat template into the file’s metadata, and a converter that does not understand a newer template writes a default one instead. The server’s startup log printed the template it had loaded. Farhan could read it, and he could see it was a template. Whether it was the template the model had been trained on was a comparison he had no reference for, because he did not have the original repository on that machine.

Friday morning, with the batch on Monday

Two routes, and they cost different things.

Patch it. Pass the missing stop strings explicitly in every request, and set max_tokens to four hundred as a backstop. Twenty minutes of work. The runaway stops immediately. The risk is that the template may also be subtly wrong, which would not look like a bug; it would look like the model being a bit worse at physics than it was in October, and nobody would be able to say when it had changed.

Convert it themselves. Download the original 16 GB of safetensors, convert to a 16-bit GGUF, quantise to the same bit width, verify the template and the stop tokens against the source, and run the same five prompts through both at temperature zero to confirm they are recognisably the same model. On the institute’s connection the download alone was an overnight job, the conversion and quantisation about forty minutes, and the verification an hour. Friday night and most of Saturday, during an admissions week in which Farhan was also the person who fixed the biometric attendance machine.

The head of the institute asked the only question that mattered: if we patch it and something is wrong with the template, when do we find out? Farhan’s honest answer was that they would find out from students saying the assistant had got worse, which is the worst kind of bug report, because by then four hundred people have a view about the tool.

The bottleneck at Shreeya Diagnostics was not the scan. It was the typing. Radiologists dictated findings into a handset; three typists across two shifts transcribed them; a radiologist read the typed version, corrected it and signed. The median delay from dictation to signature was thirty-one hours, and almost all of it was queue.

The proposal from their software vendor was to transcribe the dictation automatically and lay it out in the house report format, so the radiologist received a draft to correct and sign rather than a blank page. Nobody involved suggested the model should produce a finding or that anything would reach a patient unsigned. The whole value was in removing typing from the critical path.

The arithmetic that did not decide it

The technical director, Anjali, did the rented-per-token arithmetic first, because it is cheap and it eliminates most cases. Across eleven branches they generated about nine thousand minutes of dictation a month. Transcribed and laid out, that came to roughly twelve million tokens of input and four million of output: a few hundred rupees a month at aggregator prices, against a 24 GB card that would cost eighty thousand rupees and draw power in a room already fighting the heat.

Renting was not close. It was something like eight times cheaper and it required nothing to operate.

It was also not available. Their contract with the tertiary hospital whose outpatients they scanned named the data as confidential and restricted where it could be processed, and a recording of a radiologist describing a named patient’s chest is about as identifiable as clinical data gets. Anjali wrote the arithmetic into the decision memo anyway, with the sentence her chief executive later quoted back at her: we are choosing to spend eight times more, and here is exactly how much more, so that nobody in two years thinks this was free.

The decision inside self-hosting

That settled where it ran and opened the real question: one machine or eleven.

One server at head office. A 24 GB card running speech recognition with voice-activity detection in front of it, and an 8B model at four-bit for the layout and formatting pass, served with continuous batching so eleven branches share it. Anjali’s arithmetic said the card would sit at perhaps six per cent utilisation, which is a poor use of eighty thousand rupees and completely irrelevant, because the alternative was not a cheaper card. The problem was the link: four of the eleven branches were on connections that dropped for hours in the monsoon, and a branch that cannot reach head office falls back to typing, which is the situation they were trying to leave.

A small box per branch. No GPU. A 3B model at four-bit and a fast reimplementation of the speech model at eight-bit integer, both on the CPU. Anjali predicted the speed before buying anything: the branch machines had about fifty gigabytes a second of memory bandwidth and the model file was two gigabytes, so the ceiling was twenty-five tokens a second and the realistic figure fifteen to eighteen. A four-hundred-token report draft would take under half a minute. That is not fast and it is very far inside thirty-one hours.

Eleven boxes cost less in total than one good card. They also meant eleven machines to patch, eleven model files to keep at the same revision, and eleven places for someone to notice that the quality had changed.

What made the smaller model defensible

The 3B was worse than the 8B at the layout task and Anjali could say by how much: on forty real dictations, ninety-one per cent of drafts needed no structural correction against ninety-six per cent for the 8B. Both numbers were inside the noise of a forty-item set and she said so in the memo. What mattered more was the failure mode. The smaller model’s errors were clerical (a heading in the wrong order, a measurement left in the wrong section) and visible to the radiologist reading the draft. Neither model was asked to decide anything, and both drafts went to a person who signed.

The helpline received around four hundred messages a week, rising to twelve hundred in the fortnight around end-semester examinations. Four counsellors read everything in arrival order. The project had one purpose: read each message as it arrived, put it in one of five queues, and push anything in the urgent queue to a counsellor’s phone immediately rather than at the top of the next hour.

Nothing was to be answered by a model. The output was a queue name and a confidence, and a person read every message regardless. Lakshmi, the postgraduate student building it under the counselling head’s supervision, had been explicit about that in the one-page design, because the first question from the ethics committee had been whether a machine would be replying to a distressed student, and the answer was no.

The failure that appeared on the second day of testing

She ran a hundred and twenty real messages, anonymised, through an 8B instruct model with a clear classification prompt. It worked on ninety-odd of them.

On the messages that mentioned self-harm, it refused. Not a wrong label: a refusal: a paragraph declining to engage and suggesting the person contact a professional. The model was being asked to label a message, and it was reading the message as a request and declining it.

Those messages are the entire reason the urgent queue exists. A triage step that fails exactly on the inputs it was built to catch is worse than no triage step, because the system is now unreliable in a way that correlates with severity. In the arrival-order system, a message mentioning self-harm was read by a counsellor within the hour. In the new system it would fall out of the classifier and, unless somebody had thought about it, land nowhere.

What was available, and what it cost

A modified model. A search of the public hub turned up several copies of the same base with refusal behaviour stripped out. They were free and available that afternoon. They also had no evaluation, no dataset statement and no accountable maintainer, and they refuse nothing at all, which in a product used by distressed students is not a neutral property. The counselling head’s position was that she could not explain that choice to a parent, and that was the end of it as a serious option.

Context in the system prompt. Lakshmi tried it: a system message naming the professional setting, stating that the output is a routing label read by a qualified counsellor, and that the message is being categorised rather than answered. Refusals on the self-harm messages fell from every one to about one in six. Cost: twenty minutes.

A different model. She ran the same hundred and twenty through three other open models of similar size. The refusal rates on the same messages were wildly different: one family refused twice as often as another on identical inputs. Cost: an afternoon, and it identified a model that refused four of the relevant messages rather than nineteen.

Constrain the output. Force the answer to be one of five queue names plus an unclear value, so a refusal paragraph is not a token sequence the model can produce. Cost: an hour. This removed the failure mode entirely for the classifier, though it did not remove the underlying tendency: it made it come out as unclear instead of as prose.

Their own tuning. Three hundred labelled messages from the counsellors, a small adapter, and a model with a routing behaviour the university had written and could explain. Cost: a counsellor’s time to label, two to three weeks, and it would land after the examination peak.

The decision

The peak was five weeks away and the counselling head had to choose what ran during it. The argument that decided it came from one of the counsellors, who pointed out that the question was not which model to trust but what happens to a message the system is unsure about. If unclear went to the front of a human queue rather than the back, the worst case of the new system was the old system.

The product worked like this: a vet finished a consultation, spoke a note into the tablet, and the software produced a structured visit record (species, presenting complaint, findings, treatment given, follow-up date) which the vet checked and saved. The structuring was done by a 4B model with a carefully written prompt, eight examples in the context, and a JSON schema.

It was good and it was expensive to run in a way that mattered. The eight examples came to nine hundred tokens on every request, and with forty clinics and roughly sixty consultations a day each, that was a large and entirely repetitive bill. Wanjiru, who had built it, made the standard argument for fine-tuning: put the examples into the weights, drop the prompt to three sentences, and the per-request cost falls by most of a factor of four.

She did it properly. Six hundred real notes pulled from consenting clinics, each structured record corrected by a veterinary nurse who had done the job for years. A held-out set of a hundred she touched once. A LoRA at rank sixteen on all the linear layers, two epochs, an hour on a free notebook GPU.

The result was a tuned model that scored eighty-seven per cent on the held-out set against seventy-eight for the base model with her best prompt. Nine points, measured at identical settings, on real notes. The prompt shrank from nine hundred tokens to forty.

The second measurement

Wanjiru had built a fifty-item regression set at the start of the project, because a colleague had insisted, and she had not expected to need it. Ten general instruction-following items, ten in Swahili, ten asking for output in a format other than the trained one, ten refusal items, and ten longer notes than anything in training.

The tuned model lost fifteen points on it.

The damage was specific and, once seen, obvious. Swahili degraded most: there was no Swahili in the six hundred training notes, because the clinics that had consented to share records were the ones that wrote in English. Several clinics in the west of the country dictated in a mix of Swahili and English and the tuned model handled them visibly worse than the base had. Worse still, it had learned that the answer to anything is a JSON object: asked to write a plain-language discharge summary for a client, a feature two clinics used daily, it produced a visit record.

The deadline

A national rabies-vaccination campaign was eleven days away. Campaign weeks triple consultation volume, which was precisely when the token saving was worth most and precisely when the software could least afford to be strange.

Ship the tuned model. Nine points better on the thing they measured in the pitch deck, a four-fold cut in prompt cost during the busiest fortnight of the year, and a known, unmeasured degradation in Swahili and in the discharge-summary feature, for clinics that had not been asked and would not know why.

Ship the base model with the long prompt. Nothing breaks, nothing improves, and the bill through the campaign is what it always was.

Re-run with general data mixed in. Add twenty per cent general instruction-following examples and a hundred Swahili notes gathered from two clinics that would give them, then re-measure both sets. Wanjiru estimated four days including the labelling, which left a week of margin and assumed nothing went wrong.

Her co-founder’s position was that a nine-point gain is the number a customer understands and the fifteen-point loss is a number no customer will ever ask about. That is true, and it is exactly why the second number has to be reported alongside the first: the only person who will ever measure it is the team that made the tune.

The office received about nine hundred written grievances a month through a web form and a counter. Each had to be classified by department, assigned, and answered within a period the citizen charter fixed at fifteen working days. The backlog was structural: two staff read everything, and the reading was the queue.

The tender specified an open-weights model so that nothing left the department’s own servers. Three vendors responded and all three proposed the same 32B model, which sat near the top of a widely read public ranking and had a launch post with a large table of benchmark scores. The evaluation committee had read the ranking. One member had printed it.

Biraj, the technical officer, was asked to confirm the choice. He spent an afternoon instead.

What he built first

Forty real grievances from the previous quarter, anonymised: twenty-five in Assamese, fifteen in English, including five that the office had mishandled and two that should have been escalated rather than answered. For each he wrote the correct department and a one-line rubric for an acceptable reply. Fifteen he would use to tune prompts, twenty-five he would touch exactly once.

Then he read the launch post with six questions in front of him. Which harness, at what commit: not stated. How many shots: not stated. Single sample or aggregated: not stated, and one of the headline figures carried a small superscript that resolved to a majority vote over several samples, sitting in the same column as other models’ single-sample numbers. Who ran the baselines: three of them were copied from other papers. The language list ran to forty-one languages; the evaluation covered four, and Assamese was in neither the four nor the forty-one.

The post was not dishonest. It was simply not evidence about his task.

The measurement

He ran three candidates against the same forty items at temperature zero through one OpenAI-compatible script: the 32B everybody had proposed, an 8B from a different family, and a 12B in between. He tuned a prompt for each on the fifteen development items, because comparing a model with a prompt written for another one compares a tuned system against an untuned one.

On the twenty-five held-out items the 32B scored eighty-eight per cent and the 8B eighty-four. He wrote the noise figure into the report before the scores: with twenty-five items, a four-point gap is nothing.

The numbers that did separate them were elsewhere. The 8B’s tokenizer spent about 2.4 times fewer tokens on the Assamese grievances than the 32B’s, so the same document cost less and fitted better. At the department’s volume, the annual difference in rented cost was roughly five to one. And at a realistic concurrency, the 32B’s p95 time to a complete draft was eleven seconds against three; the citizen-charter target is measured in days, so that mattered less than it looked, but the counter staff waiting for a draft while a citizen stood in front of them disagreed.

The failure column was the useful part of the report. The 32B invented section numbers of the Assam Panchayat Act when a grievance mentioned one. The 8B did not, because it declined more often, which was worse on two English items and better on six Assamese ones.

The part that was not technical

Biraj’s recommendation was the 8B. The committee had a printed leaderboard on which the 8B did not appear in the top twenty, and a procurement culture in which choosing the lower-ranked thing requires a defence that choosing the higher-ranked thing does not.

The vendor proposing the 32B pointed out, correctly, that a four-point gap favoured their model. Biraj’s counter was that the gap was inside the interval on twenty-five items and the cost difference was not.