Fine-Tuning, and When Not To

Ravi ran operations at a freight forwarder in Pune. Customers wrote in asking where a consignment was, why a duty figure had changed, whether a delivery could be rescheduled. A hosted model drafted replies. It was adequate and it was wrong often enough that two people read every draft before it went out.

In March he took a proposal to the two directors: fine-tune a model on eighteen months of the company’s own correspondence, so that it would answer “the way we answer”. The directors liked it. One of them repeated the phrase “our own AI model” to a customer the following week. A budget of four lakh rupees and six weeks was approved.

The engineer Ravi hired, Sneha, began where the course says to begin. She pulled thirty replies the drafting model had got wrong (real ones, from the archive, not ones anybody invented) and sorted them by what she would have had to change first.

Nineteen of the thirty were knowledge failures. The model did not know the current duty schedule, did not know that one customer had a negotiated rate, did not know which of three warehouses a consignment had been routed through. Six were format: the draft opened with an apology the firm never used, or put the tracking number at the bottom. Three were adherence: the draft ignored two of the four instructions in the prompt. Two were genuine capability failures involving a multi-leg date calculation.

Then she ran the paste test on the nineteen. She put the relevant document into the prompt (the duty circular, the rate card, the routing record) and asked again. Seventeen of the nineteen came back correct.

That is the whole finding, and it was available in a day and a half.

Sneha wrote it up plainly: the firm’s problem was that the model could not see its own records. Retrieval over the rate cards, the duty circulars and the shipment database would fix seventeen of thirty failures. Constrained decoding and five examples in the prompt would fix the six format ones. Fine-tuning was the right tool for perhaps three of the thirty, and for those three it would need a dataset, an evaluation set, and somebody to rebuild it every time the base model moved.

She also costed the version Ravi had proposed. Two thousand examples at five minutes each of somebody’s careful attention. A held-out set of two hundred, built first. Eight to fifteen training runs rather than one. A card to serve it on, or a provider that hosts adapters. And a standing question every six to nine months, when the base model was superseded, about whether to retrain or to stay frozen at last year’s quality.

Ravi read it and saw the difficulty immediately. The technical answer was clear. The organisational one was not. A director had told a customer the firm was building its own model. Coming back with “we are going to write a better prompt and connect the database” sounded, to anyone who had heard the first version, like a retreat.

The cost of the honest answer was a conversation Ravi did not want to have, and the visible shrinking of a project he had proposed. The cost of the other answer was six weeks, four lakh rupees, and a model that would still not know the duty schedule, because no amount of training installs a fact that changes every quarter, and training on those answers would have taught the system to state duty figures confidently whether or not it knew them.

The office had four hundred school inspection reports a month, each two to six pages of Hindi prose written by a visiting officer, and a requirement that each be reduced to a six-line summary for the block education officer. Two clerks did this. It took them most of a week.

Anjali, on contract from a state technology cell, was given a month and no budget. No budget meant a free Kaggle notebook: a T4, sessions of up to twelve hours, roughly thirty GPU-hours a week.

Her first run diverged. The loss fell for forty steps, spiked, and became NaN at step 212. She restarted it. It reached step 190 and did the same. The logs were full of lines about the gradient scaler halving its scale.

She had copied a configuration from a tutorial: QLoRA on an 8B model, learning rate 2e-4, no warmup because the tutorial’s author had removed it, sequence length 1,024. The configuration was not unreasonable. It was written for an A100.

The T4 is a Turing card and has no bf16. Everything ran in fp16, which has a maximum representable value of 65,504 and a smallest normal value around six times ten to the minus five. Gradients below that round to zero, so fp16 training multiplies the loss by a large constant before the backward pass and divides it out afterwards; and when that constant is too large, gradients overflow to infinity and the framework throws the step away and halves the scale. A healthy fp16 run discards a step occasionally. Anjali’s was discarding them every few steps, and then diverging outright.

She had three ways forward, and a deadline in eleven days.

The first was to pay. A rented 24 GB card with bf16 costs somewhere around fifty rupees an hour on a community marketplace, and her whole run was perhaps two hours. The obstacle was not the money, which was trivial; it was that the office had no mechanism to spend it. A purchase of any size needed a sanction that took three weeks.

The second was to lower the learning rate and keep the free card. Halving it to 1e-4 and adding ten steps of warmup would very probably stabilise the run. It would also mean the adapter travelled less far in the same number of steps, and she would need more steps to reach the same place, on a card that was already the slow part.

The third was to use a smaller model. A 1.5B model in fp16 with a LoRA fits a T4 comfortably at sequence 2,048, trains several times faster, and at that size full fine-tuning is affordable too. The summaries would be less fluent. Hindi tokenisation on the 8B model she had chosen was already producing about three tokens per word, which was eating her sequence budget; a model with better Devanagari coverage would help more than any hyperparameter.

She ran the cheap experiment first, because it cost half an hour. Three runs of 150 steps each, at 1e-4, 2e-4 and 5e-4, on the 8B model, and she looked at the training loss curves rather than at anybody’s advice. At 5e-4 it spiked in the first fifty steps. At 2e-4 it fell and then rose. At 1e-4 it showed the shape she wanted: a fast drop in the first fifty steps, then a smoother decline, no scaler messages.

The hospital had one constraint that removed most of the options: patient notes could not leave its premises. No hosted model, no rented card in somebody else’s data centre, no batch endpoint. Whatever ran, ran on a single 24 GB workstation card that the radiology department had bought for something else and was willing to share overnight.

Dr Menon, who ran the project with one contract engineer, had three thousand dictated notes with a matching discharge summary written by a registrar. The task was narrow and well specified: take the dictation, produce eleven fields in a fixed order, in the hospital’s house phrasing, with the medication list formatted as the pharmacy wanted it.

The engineer, Faiz, laid out three options on one page, with the memory arithmetic done before anything was rented or downloaded.

Full fine-tuning an 8B model was out. Mixed precision with AdamW costs about sixteen bytes per parameter of persistent state: four for the fp32 master weights, four for the gradients, eight for AdamW’s two moment buffers. That is 128 GB before a single activation, on a card that has 24. Sharding across cards was not available; there was one card.

QLoRA on an 8B model fitted. The frozen base in NF4 is about 4.1 GB, the adapter and its optimiser state under a gigabyte, activations with gradient checkpointing a few more. Call it ten or eleven gigabytes, comfortably inside the card. The costs were real but bounded: roughly a quarter to a third lower throughput than an unquantised LoRA, because every matrix multiplication dequantises its weights on the fly; and an adapter fitted against a four-bit base, which makes merging into the full-precision base an approximation rather than an identity.

Full fine-tuning a 1.5B model also fitted. Sixteen bytes per parameter on 1.5 billion parameters is 24 GB, which is the whole card; but with 8-bit AdamW the optimiser state term falls from eight bytes to two, giving ten bytes per parameter, or 15 GB, with room for activations. Simpler: no adapters, no quantisation, no merging question at the end, one ordinary model to serve.

The decision was not obvious, and both sides had a cost the hospital would feel.

The 8B model would write better prose. Discharge summaries are read by general practitioners who did not see the patient, and fluency is not decoration there. But serving it meant the same card during the day, running a quantised model at a few requests a minute, and a merging step whose effect on quality nobody had measured.

The 1.5B model would be markedly faster to train and to serve, would run on the ward’s own machines if it ever needed to, and would be one artefact with no adapter to keep aligned with a base revision. Against that: a small model collapses rather than degrades when an input is unlike anything it saw. A dictation that wandered (and registrars’ dictations wander) would get triage-shaped output confidently.

Faiz did what the module recommends and ran the cheap version first. A QLoRA on the 8B model over the three thousand notes, two epochs, effective batch of sixteen, took under three hours overnight and told him something he had not expected: the eleven-field structure was learned almost perfectly within one epoch, and what the second epoch added was memorisation of individual registrars’ phrasings.

The non-profit answered about two thousand messages a month from parents, in Hindi, Bhojpuri-inflected Hindi, and a good deal of transliterated Hinglish. Four field staff answered them between other duties. The plan was a model that drafted the reply, with a person approving it.

Work started in June. Three volunteers wrote gold answers for real parent messages pulled from the archive, at a pace of about sixty a week each. By the third week they had 940 examples and everybody was pleased with the rate.

In the fourth week the project lead, Kavita, did the thing the module asks for and read a hundred of them from beginning to end.

They were not bad answers. That was the difficulty. Each one, taken alone, was something a careful person would have been content to send. Read together, they were four different people.

One volunteer opened every reply with a greeting and the parent’s name. One opened with the answer. One apologised when the news was unwelcome; the others stated the position. Two wrote amounts as “₹1,200” and one wrote “1200 rupees”. When the answer was not known, there were eleven different forms of words across the set, ranging from a promise to find out to a suggestion that the parent visit the office.

Lengths varied from one line to nine. Two volunteers showed their reasoning about eligibility; one gave only the conclusion. And the language mix varied with whoever had answered: one volunteer replied in Devanagari to transliterated questions, the others matched the parent’s script.

None of this was an error by any individual. It was a specification that had never been written down, which meant each volunteer had supplied their own.

Kavita understood what it would do. A model does not learn the right answer; it learns the distribution of the answers in front of it. Trained on that set, it would greet about a third of the time, apologise unpredictably, and produce one of eleven different ways of saying it did not know, which is the same as having no reliable way of declining at all. Every one of those unforced variations would become randomness in production, reproduced faithfully.

The decision was about the 940 examples.

Throwing them out meant losing three weeks of three volunteers’ evenings. The volunteers were unpaid, they had been told their work mattered, and two of them had exams in August. Asking them to redo it was asking for something Kavita was not sure she would get, and a project that loses its annotators in week four does not usually recover.

Keeping them meant training on a coin flip. It also meant that the style guide she was about to write would apply only to whatever came next, so the dataset would be half one thing and half another, which is the mixture that teaches a model that the system prompt is optional and the greeting is a matter of chance.

There was a third option she nearly took, which was to keep the 940 and add two thousand consistent ones on top, on the theory that the consistent majority would win. She talked herself out of it by working out what it actually said: that she would pay for two thousand more examples in order to drown out nine hundred she already had and could simply correct.

The product let a head teacher type “how many girls in class eight have not paid the second term fee” and get a table. Behind it, a model wrote SQL against a fixed schema of forty tables, the query ran against a read-only replica, and the rows came back.

It worked about seventy-eight per cent of the time. The failures were not syntax (constrained decoding had taken care of that months earlier) but queries that ran cleanly and returned the wrong rows: a join through the wrong key, a term filter omitted, a count of enrolments where a count of pupils was wanted.

The team had something most teams do not: a real verifier. For eleven hundred questions they had a hand-written reference query and its result set on a test database. A generated query could be executed and its rows compared. Correct or not correct, instantly, at no cost, unlimited times.

That made reinforcement learning with a verifiable reward genuinely available, and the engineer leading it, Prakash, wanted to use GRPO. The argument was reasonable. Sampling eight completions per question and scoring them against the reference gives the group its own baseline, removes the value model that makes PPO expensive, and is the method behind the recent open reasoning models.

His colleague Divya proposed the dull alternative first: rejection-sampling fine-tuning. Generate eight completions per question with the existing model, keep only those whose result set matches the reference, train the model on those with ordinary supervised fine-tuning. No reinforcement learning, no reward function to tune, no generation loop inside a training loop.

The cost on each side was concrete. GRPO meant generating eight responses per prompt at every step, inside training, which put the run in GPU-days rather than hours; on a rented A100 that was the best part of a week of compute and, more importantly, two weeks of Prakash’s time on a method neither of them had run before. Rejection sampling meant one generation pass over eleven hundred questions (a few hours) and then a training run they had done ten times.

Divya’s argument was the one the module makes: rejection-sampling fine-tuning on verified outputs is the baseline you should beat before you reach for any of this.

Prakash’s counter was that the interesting failures were the ones the model never got right in eight samples, and rejection sampling learns nothing from those: it simply discards them.

That turned out to be the testable question, and it took an afternoon. They sampled the base model a hundred times on fifty held-out questions and counted how often the correct query appeared at least once. For thirty-nine of the fifty it did. For eleven it never did, across a hundred attempts.

That number decided a great deal. If a method that sharpens a model’s distribution towards paths it can already sometimes find is what reinforcement learning on verifiable rewards mostly does (and whether it teaches new capability or elicits existing capability is genuinely argued, with careful experiments on both sides) then the eleven questions the base model never solved were unlikely to be rescued by it either.

They also wrote down, before running anything, what a reward hack would look like. A query that returns an empty result set matches a reference whose answer is legitimately empty. A query that selects everything and lets the comparison sort it out. A query with the answer hard-coded as a literal, which is the SQL equivalent of returning 4181 when the test asks for the seventeenth term.

The helpline took calls and messages about land record certificates. A model had been fine-tuned to draft the written replies, and it was good: the register was right, the document names were right, the steps were in the order the counter clerks used.

It also, on roughly one reply in three, answered the question, then wrote a new question from the citizen, then answered that one too. Sometimes for four turns. The text was fluent throughout and increasingly invented.

The team had three days. The launch date had been in a press note.

Their first fix took twenty minutes and worked. The serving layer accepted a stop string, so they set it to the sequence that preceded a user turn, and the extra conversation disappeared from every response. They demonstrated it to the programme director on a Tuesday afternoon and she was satisfied.

The engineer, Imran, was not, and he said so in a way that made the next two days awkward. A stop string at the API layer hides a symptom. Whatever had taught the model to continue was still in the weights, and it would surface somewhere else: in a reply that trailed off mid-sentence because the stop string fired early, in a long answer that ran to the token limit, in the next fine-tune built on the same pipeline.

He spent an hour on the four checks the course prescribes before any run, which nobody had done before the original run.

The first check printed one templated example. It looked correct.

The second check was the one that answered it. He decoded the unmasked labels of a real batch (the tokens the model was actually being scored on) and printed them beside the masked ones. The trained region was the assistant’s reply, correctly, and it ended one token early. The end-of-turn token was in the masked region.

The cause was two lines of ordinary setup. The base model shipped without a pad token, as many do, so somebody had written the near-universal workaround of setting the pad token equal to the end-of-sequence token. The collator then built its label mask by matching on the pad id. Every position holding that id was masked, including the real end-of-sequence token at the end of every answer.

The model had therefore never once received a gradient teaching it to stop. It had been trained, thousands of times, on answers that simply ended, with nothing marking the ending as something to produce. At inference it did the only thing it had been shown: it kept going.

The loss curve had been healthy throughout. Nothing in the training logs was wrong. The bug was invisible in every number the run produced and visible in five seconds of decoded labels.

Imran’s proposal was to add a distinct pad token, resize the embeddings, rebuild the mask from the attention mask rather than from the token id, and retrain. The run itself was two and a half hours. The evaluation was another half day. Realistically it put the launch two days late.

Against that, the director’s position was not unreasonable. The stop string worked. The demonstration had been fine. A press note had a date on it, and a delay would have to be explained upwards.

The practice had two articled clerks keying invoice data into accounting software: supplier name, GSTIN, invoice number, date, taxable value, tax split, total. Most invoices arrived as PDFs by email, some as photographs of paper taken on a phone, and a few hundred a month as scans of handwritten delivery challans from clients in the textile trade.

The partner who had read about it, Nilesh, wanted to fine-tune a vision-language model. The appeal was obvious: one model, any image, out comes structured data. He had found a recipe, and a rented card for a day was affordable.

The engineer the practice retained, Priya, asked for an afternoon before anything was trained, and ran the comparison the module recommends. Fifty pages, drawn deliberately: thirty machine-printed PDF invoices, ten phone photographs of printed invoices, ten scans of handwritten challans.

Pipeline A was an OCR engine with layout output feeding a text model that had been given five examples in its prompt and a constrained decoder enforcing the schema. Nothing was trained.

Pipeline B was an off-the-shelf vision-language model, prompted, also with a constrained decoder. Nothing was trained.

On the thirty printed PDFs the OCR pipeline extracted all seven fields correctly on twenty-nine. The vision model managed twenty-five, and three of its four failures were the GSTIN: a fifteen-character alphanumeric string where a single wrong character makes the record useless, and where reading is exactly what an OCR specialist is built for.

On the ten phone photographs they were close: OCR eight, vision model eight, failing on different pages.

On the ten handwritten challans the OCR pipeline extracted almost nothing usable. The vision model got six.

Priya also measured something Nilesh had not asked about, and it changed the shape of the decision. She encoded one full-resolution invoice page and printed how many sequence positions it consumed in the vision model. At the resolution needed to read the tax table, the page occupied well over a thousand positions, and the model tiled larger images, multiplying that. Activation memory scales with sequence length, so a training batch of four pages needed more memory than a batch of thirty-two text examples. Her effective batch would be one or two per device with heavy accumulation, and the run Nilesh had budgeted a day for was closer to three.

So the choice was not between two tools. It was between one architecture and two.

Fine-tuning the vision model on everything meant one artefact, one evaluation, one thing to maintain, and worse accuracy on the eighty-five per cent of the volume that was printed, at several times the cost per page, with failures that are hard to inspect because you cannot look at an intermediate output and see where the reading went wrong.

Running both pipelines meant two things to maintain, a router deciding which page goes where, and the clerks seeing two different failure modes. Against that it put a specialist on each job: the OCR engine on printed text, where it is simply better, and the vision model on handwriting, where layout and stroke carry meaning that survives no text extraction.

The number on the slide was 0.94. Routing accuracy, six queues, on a held-out set of six hundred messages. The base model with the best prompt the team could write had scored 0.86. The project had run for eleven weeks and the figure had been shown to the group’s medical director twice.

Four days before the launch, a data analyst called Meera was asked to write the evaluation section of the model card. She needed to state how the held-out set had been constructed, and discovered that nobody had written it down.

The split had been random. Six thousand messages, shuffled, ninety per cent for training and ten per cent held out.

She ran three checks, each of which took under an hour.

The first was near-duplicate overlap across the split. She built a MinHash index over the training texts and queried every evaluation item against it. Eighty-one of the six hundred evaluation messages had a near-duplicate in the training set: thirteen and a half per cent. Patients ask the same things in the same words, and appointment-rescheduling messages in particular are close to templated.

The second was group leakage. The messages carried a patient identifier. Two hundred and six of the six hundred evaluation messages came from patients who also appeared in training, some of them with a dozen messages each. The model had learned individual patients’ phrasing, their conditions, the department they usually wrote to.

The third was time. The data ran from January to August. A random split had put August in training and February in evaluation, which is the reverse of what production does. In May the group had merged two departments and renamed a queue.

She rebuilt the split properly: grouped by patient, cut at a date so that everything after the first of July was evaluation and everything before was training, and removed evaluation items with a near-duplicate on the other side.

The model scored 0.79.

The prompt baseline, re-run on the same corrected set, scored 0.77.

That is the whole finding. Eleven weeks of work had bought two points, not eight, and on a corrected set of four hundred and eleven items the ninety-five per cent interval on a difference of two points comfortably includes zero.

The decision Meera put in front of the project lead was not whether the model was good. It was what to say, four days before a launch, about a number that had been shown to the medical director twice.

Launching on the old number meant a figure in a model card that the team now knew was inflated by leakage, in a clinical setting, where the queue a message lands in determines how quickly somebody reads it. Correcting it meant explaining that the headline figure had been wrong for eleven weeks, that the honest gap over a prompt was within noise, and that the launch should probably be a pilot rather than a launch.

There was also a third thing, which Meera raised and which nobody had thought about at all. Nobody had run a refusal set. The messages included requests for medical advice that the system was never intended to answer, and no one had measured whether the fine-tuned model still declined them as the base model had.

The centres had tablets, two gigabytes of usable RAM each, and no connectivity worth depending on. The product was a model that answered physics and chemistry doubts in the chain’s own explanatory style (a fixed structure of restate, principle, working, common mistake) which the teaching staff had spent years standardising and which the students recognised.

The team fine-tuned a 1.5B model. At that size full fine-tuning was affordable and simpler than an adapter, and the results were good. On four hundred held-out doubts the tuned model produced the four-part structure on 96 per cent of answers against 31 per cent for the base model with the best prompt, and two senior teachers preferred its explanations in a blind comparison at better than three to one.

Then it was converted for the tablets: merged, converted to GGUF, quantised to Q4_K_M, which brought the file to about a gigabyte and made it fit.

The first field build came back from two centres with the same complaint. The answers were fine but they were not the chain’s answers. No four-part structure. The common-mistake paragraph, which the teachers considered the most useful part, was simply absent most of the time.

The engineer, Tanvi, assumed a template problem and spent a day on it. The Modelfile template was correct and matched training. The prompt was identical. The tokenizer was the one saved beside the weights.

What she had not done was evaluate the artefact she shipped. Every number in the project came from the fp16 model on the training rig.

She ran the three-way comparison, which took twenty minutes: base quantised, tuned fp16, tuned quantised, on the same four hundred doubts. The tuned fp16 model produced the structure on 96 per cent. The tuned quantised model produced it on 44 per cent. The quantised base produced it on 29 per cent.

The fine-tune had very nearly been rounded away.

The mechanism is unglamorous. Her training had changed the weights by a small amount: full fine-tuning of a small model over four thousand examples is still a modest perturbation. Quantisation error at four bits is roughly constant per weight, and on a 1.5B model there is much less redundancy to absorb it than on an 8B one. When the quantisation error is of comparable size to the change training made, the change is rounded away, and the quantised tuned model behaves substantially like the quantised base.

Her options each cost something the chain would feel.

Ship at Q8_0. About 1.8 GB instead of 1.0, which on a tablet with two gigabytes of usable RAM does not leave room for the application. Some centres had newer tablets; most did not.

Ship a larger model at four bits, where quantisation costs proportionally less. A 3B model at Q4_K_M is around two gigabytes, which is worse on the same tablets, and slower on processors already producing a few tokens a second.

Train harder, so that the delta is larger than the quantisation error. More epochs and a higher learning rate would push the weights further, and on a 1.5B model that also means more memorisation and a sharper collapse on doubts unlike anything in the training set, which in a subject where students ask strange questions is not a small cost.

Accept 44 per cent, and have the application enforce the structure instead.