Running Models Yourself

The registrar had read that an 8B model at 4-bit runs on a card costing about half a month’s salary, and that it summarises well. Both claims are true. He wrote a note asking for the money.

The hospital’s medical superintendent asked one question before signing: how do we know it is good enough at this. Nobody could answer, because nobody had tried it. The obvious way to try it is the audition: take thirty real items, run them through a hosted copy of the exact open weights, and grade the outputs. It costs under a dollar and it answers the model question without answering the hardware question.

Except that the reason for the project was that ward notes cannot leave the building. Running thirty real discharge notes through a hosted endpoint, even for an hour, even for a test, is exactly the thing the whole deployment exists to prevent. The registrar had proposed to establish privacy by breaching it.

The two ways out, and what each costs

Write synthetic notes. Invent thirty plausible ward notes and audition on those. Free, immediate, and nobody’s data moves. The problem is the one the course states plainly: invented examples are always easier than real ones in a specific way. They are shorter, cleaner, better punctuated, and free of the internal shorthand that actually breaks models. Ward notes at this hospital contain abbreviations used only in that ward, drug names written three ways, Marathi words in Devanagari inside English sentences, and handwriting transcribed by whoever was on duty. A synthetic set would have contained none of that, and the audition would have returned a clear pass on a task the model had not been asked to do.

De-identify thirty real notes by hand. Two people, a morning each, removing names, registration numbers, addresses and dates, and checking each other’s work. Real structure, real abbreviations, real mess, and nothing identifiable leaves. The cost is a day of two clinicians’ time before a single line of the project exists, spent on something that produces no deliverable, which is precisely the kind of work that gets cut when a project is trying to prove itself quickly.

There was a third option nobody liked: buy the card first and audition locally. It removes the privacy question entirely. It also spends the money before learning whether the weights can do the job, which is the sequence this course exists to prevent.

What the set was for

The registrar argued for synthetic notes on the grounds that the model would be corrected by a doctor anyway. The superintendent’s objection was not about the risk of a bad summary. It was that an audition on easy items cannot distinguish between two models, which means it cannot tell you which weights to shop for, which means the purchase is still a guess with a document attached.

They also had to decide what the thirty items were for afterwards. The course is insistent that the set is the most valuable artefact the project owns: it is what tells you later whether a quantised copy still works, whether a new release is better, and whether a prompt change helped. A de-identified set can be kept, versioned and re-run for years. A set of real notes cannot be kept at all.

That argument turned out to matter more than the audition itself.

The centre’s systems person had done the arithmetic correctly, which is why the problem was interesting. A 14B at Q4_K_M is about 8.4 GB of weights. The model’s config file gives 48 layers, 8 KV heads and a head dimension of 128, so the cache costs two bytes for K and V, times 48, times 8, times 128, times two more for fp16: about 192 KiB per token. At the 16,384-token window he wanted, that is 3.0 GB. Add a gigabyte of overhead and the total is 12.4 GB on a 12 GB card.

It did not fit, by four hundred megabytes.

What happened instead is the part worth studying. llama.cpp did not refuse. It placed forty-four layers on the card and four on the CPU, and the assistant worked. In testing, with one person asking questions, it produced about eleven tokens per second where the same model fully resident produced twenty-six. Four layers out of forty-eight (eight per cent of the model) had cost well over half the speed, because every single token has to pass through all forty-eight of them and waits on the slow four.

Three options, priced

Drop to an 8B. Free, immediate, and about twice as fast because half as many bytes are read per token. The cost is real and it is exactly where this centre could least afford it: the questions students ask at half past ten at night are multi-step mechanics problems, and multi-step reasoning is the most size-sensitive thing a model does. The 8B answered them fluently and got the third step wrong often enough that a teacher would have to check every answer, which removes the point.

Keep the 14B and cut the window to 8k. The cache halves to 1.5 GB, the total comes to 10.9 GB, everything fits, and the speed returns to twenty-six tokens per second. The cost is that the 16k window existed for a reason: the assistant was pasting whole chapters of the textbook into the prompt, because nobody had built a retriever. Cutting the window means building one, which the systems person estimated at a fortnight he did not have before the session started.

Keep the 14B and the window, and quantise the KV cache to 8 bits. This halves the cache term to 1.5 GB for a quality difference most tasks cannot detect, and it requires flash attention to be switched on or the flags are silently ignored. Total 10.9 GB, fully resident, twenty-six tokens per second, window intact. Roughly ten minutes of work.

The third option looks obviously correct written down like that, and it was not obvious in the room, because the argument in the room was about which model to use. Both of the first two options are answers to the question which model. The arithmetic says the oversized term was not the weights at all: the cache was a quarter of the budget and the weights two thirds, so halving the cache saved an eighth of the total and that was enough.

The part that came back later

There was a fourth consideration nobody raised on the night. The centre’s plan was eventually to let several students query at once, and each concurrent sequence needs its own cache. A configuration that fits exactly one conversation at 16k fits precisely one student, and the second arrival would have pushed it back into offload: the failure that arrives not at startup but when the service gets popular.

The desk used the model for two jobs. It cleaned up English wire copy before translation, and it proof-read Marathi copy filed by district correspondents. The first job was reported as excellent. The second was reported as fine, then as slightly odd, then as something two sub-editors had quietly stopped using because it was faster to do it themselves.

Nobody had measured anything. The model had been chosen on an English benchmark table, the quant had been downloaded because it was the one the community had published, and the evaluation had been a week of people trying it.

The chief sub-editor did the one thing that settles this class of argument: she built a set. Thirty pieces of copy, twenty in Marathi and ten in English, each with the errors marked by hand in advance. She ran them at temperature 0 against the 4-bit file they were using and against a hosted copy of the same model at full precision.

The English results were identical to within one item. The Marathi results were not close. The 4-bit file missed nine errors the full-precision model caught, and four of the nine were in Devanagari conjunct spellings: precisely the rare, precision-critical tokens the course warns about.

Why, and what it costs to fix

The mechanism is the importance matrix. A low-bit quant is produced with a matrix computed by running the full-precision model over a body of calibration text and recording which weights are multiplied by large activations. Those weights get protected; the rest are rounded harder. Whatever the calibration text did not exercise was fair game, and the quantiser did exactly what it was told.

The community quant they were using had been calibrated on a general English corpus with some code in it. The weights that matter for Marathi were never exercised, so they were not protected, and the damage landed there first. In English nothing had happened, which is why three months passed without a complaint from anyone reading English.

Two remedies, and both cost something.

Buy a second-hand 16 GB card and run Q6_K. At six bits the damage largely disappears and no calibration question arises. The desk had no budget line for hardware, and the purchase would have to be argued through a finance process that treats capital differently from a monthly subscription. That argument was estimated at two months and an uncertain outcome.

Produce their own importance matrix, calibrated on Marathi. Free. The tooling is llama-imatrix followed by llama-quantize, and the desk had three years of its own published copy to calibrate on. The cost is that a quant calibrated on Marathi is better at Marathi and worse at everything else, and the same desk also handles English wire copy. They would be trading one of their two jobs for the other.

There was a third option that nobody had considered because it does not involve quantisation at all: find a base model whose training covered Marathi properly, and check whether a smaller one of those beats a larger general model at four bits. Language coverage is a property of the weights, and no quantisation choice repairs a model that was thin in a language to begin with.

The teacher had installed Ollama on one laptop over a weekend and pulled a 4B model. It worked. It answered questions about a comprehension passage, it was private, it needed no internet during the lesson, and it was slow in a way she assumed was simply what a laptop does.

Two tokens a second is not what that laptop does. It is what that laptop does when the model is on the CPU and nothing else.

The laptops had Intel integrated graphics. The installer had picked a build with no Vulkan backend compiled in, so every layer ran on the processor. Nothing reported an error, because nothing had gone wrong from the software’s point of view: a CPU fallback is a supported path, and it is the one everything silently lands on.

There was a second problem underneath it. The comprehension passages were about nine hundred words, and the follow-up discussion added the class’s questions on top. Ollama allocates a modest context window by default regardless of what the model supports, and the older build on those machines allocated 2,048 tokens. By the fourth question the beginning of the passage had been dropped, with no error and no warning, and the model was answering about a text it could no longer see. The teacher had recorded this in her notes as the model getting confused, which is how it looks.

The decision

A volunteer from a nearby engineering college diagnosed both in about forty minutes, and then the decision was not a technical one.

Fix it properly. Install a Vulkan build on all twelve machines and raise the context window. The Vulkan backend reaches Intel and AMD integrated graphics, and on this class of machine it triples prompt processing while doing nothing at all for generation, because an integrated GPU shares the same system memory and therefore the same bandwidth ceiling. Prompt processing was the problem: a nine-hundred-word passage was forty seconds of silence before the first word appeared. The cost was that the laptops were locked down. Installing anything required an administrator account held by the district IT office, and the process for that ran in weeks.

Use a smaller model. A 1.7B at Q4 reads about 1.1 GB per token instead of 2.5, so it roughly doubles the generation rate on a bandwidth-limited machine, and it needs no new build and no administrator. The cost is what a 1.7B is: reliable at short, well-specified jobs and unable to hold more than about three instructions. The comprehension exercises the teacher wanted involved asking the model to compare two paragraphs and justify an answer, which is two steps past what that size class does.

The volunteer pointed out a third thing, which was free and which neither option covered. Whatever they did about the build, a nine-hundred-word passage sent on every single question was being re-processed every single time. Sending the passage once and keeping the conversation on the same slot would remove most of the wait without changing a single component.

The assistant had been running for a term and had been fine, because for a term it had been used by four teaching assistants during the day. Then it was announced to the students, and the first evening lab began at seven o’clock with about three hundred people opening the page within ninety seconds of each other.

What the students experienced was not an error. The page accepted their question, showed a spinner, and eventually either answered or timed out in the browser. Several of them pressed the button again. By quarter past seven the queue held more than two thousand requests and the card was generating answers for people who had closed the tab twenty minutes earlier.

This is the standard failure shape and it arrives quickly. An unbounded queue converts a throughput problem into a total outage: requests time out at the client, the client retries, the retries join the same queue, and the server spends its whole capacity producing output nobody is waiting for.

What the measurement said

The following week they ran an open-loop load test: a fixed arrival rate rather than a fixed number of workers, because a closed-loop test slows down automatically when the server does and therefore can never show a queue building. Prompt and output lengths were drawn from the previous week’s logs rather than invented.

The curve was the usual one. The 95th-percentile time to first token stayed flat, then bent, then went vertical. The rate just before the bend was about eleven requests a second, and the lab produced roughly forty in the first minute and then a long tail.

Two tuning changes were available and both helped. Chunked prefill stopped one student pasting a four-hundred-line traceback from freezing everyone else’s stream mid-sentence, which had been reported as the server being unreliable rather than slow. Prefix caching mattered more: every request carried the same eight-hundred-token preamble of lab rules and style guidance, and with the cache hitting, that work was done once instead of forty times a minute. The preamble had to be moved above the student’s name, which had been at the top and had been invalidating the cache on every single request.

Neither change raised the ceiling by an order of magnitude. Tuning buys a factor, not a factor of ten.

The decision

So the real question was what happens past capacity, and it was not a technical question.

Queue everything. Nobody is refused. Everybody waits, some of them for minutes, retries pile on, and the service degrades for all three hundred at once. It has the appearance of fairness and it is the configuration that produced the first evening.

Bound the queue and refuse fast. Past a set depth, return a refusal immediately with a wait time. Some students are told to come back in a minute, which is a visible, attributable failure that a first-year student will describe as the college’s assistant not working. The rest get answers at a predictable speed.

The head of department did not like the second option, for an honest reason: a refusal is something a student notices and complains about, and a slow answer is something they blame on the wifi.

The cell processed several thousand questions a month, tagging each with a syllabus unit and a difficulty band. A local 8B model did the first pass and a clerk checked it, and for five months the clerk had been changing roughly one tag in twenty. In the week after an engine upgrade she was changing about one in six.

She reported it as the model getting worse. Three people had three explanations within an hour: the quant was bad, the new engine had a sampling regression, and somebody had edited the prompt. None of them had any evidence, because nothing had ever been measured. The clerk’s one-in-twenty was itself a memory rather than a record.

The cell’s technical officer proposed to stop and build the thirty-item harness the course describes before changing anything. Thirty frozen items with the correct tag written down, run at temperature 0, scored automatically against the expected label, with latency and token counts recorded beside each one. Two days, including collecting items that covered the awkward cases: questions with diagrams described in text, questions spanning two units, questions in the physics paper that mention chemistry.

The cost of stopping to measure

Those two days were not free. The tagging stayed wrong, the clerk re-tagged by hand, and the backlog grew during an admissions period. The head of the cell’s position was reasonable: we know when it broke, so roll back the upgrade this afternoon and the problem goes away.

Rolling back would have worked. It would also have left the cell in exactly the position it was in that morning: unable to say what had changed, unable to accept the next upgrade without the same fortnight of doubt, and unable to distinguish a real regression from a bad week, because a one-in-six figure remembered by one person is not a measurement.

The technical officer made the argument in the terms the course uses. Every question in the local-model subject becomes an argument without a harness and a ten-minute run with one. Is Q4 worse than Q5, did the prompt rewrite help, is the new release better, did the upgrade break something. They were about to answer one of those by guessing, and then answer the next one by guessing too.

What the diagnostic ladder found

With the set built, the ladder took under an hour and the causes were not any of the three theories.

The first rung came back clean: the card was being used. The second did not. The upgrade had changed a default context length, and the cell’s prompt carried a long list of unit definitions before the question. Under the new default the prompt was being truncated: the usage.prompt_tokens field in the response, compared against the configured window, showed it immediately. The definitions for the last four syllabus units were being dropped, and every question belonging to those units was tagged from whatever the model could still see.

That also explained a detail nobody had connected: the errors were not spread evenly across units. They clustered.

The change was one environment variable. The reasoning was ordinary: the developer needed the model from home, the office had a static address from its provider, and setting up a tunnel looked like an afternoon nobody had. The port was left open over the weekend.

The firm found out on Monday because the model list had changed. Two models nobody recognised had been pulled onto the disk, and about ninety gigabytes of the drive was gone. The server log showed requests arriving from Sunday morning onwards, in volume.

Nothing dramatic had happened, which is the usual shape. Internet-wide scanners sweep the whole address space on common ports in hours, and there are public search engines whose product is the resulting index. Eleven hours is unremarkable. Somebody pulled the model list, found a permissive model, and used the firm’s electricity to generate whatever they wanted. The management endpoints did the rest: an exposed instance of that engine allows a caller to pull new models onto the disk and to delete the ones you have.

Two things made it worse than a wasted weekend of power. The first is that the firm’s own logs now contained other people’s prompts. The second is the one that actually frightened the partners: every token produced came from their machine and their address, and abuse originating there is attributable to them before it is attributable to anyone else.

What they had not understood

The developer’s mental model was that the engine had an API key setting, so it had authentication. It does for some engines and not that one, and where it exists it is a single shared bearer token with no per-person accounting and no rate limiting. Revoking it locks out everybody. It is better than nothing and it is not an access control system.

The second misconception took longer to dislodge. Somebody suggested that nobody could have found the machine because they had never forwarded a port. The office router was handing out globally routable IPv6 addresses with a default-allow inbound rule, so the machine had been directly addressable without anybody deciding it should be. The asterisk in the listening-socket output had been there all along and meant exactly that.

The decision

The engine went back to loopback the same morning. The argument was about how the developer reaches it from home.

SSH tunnels. Free, already installed, and per-person by construction: access is exactly the set of people with a key on that machine, and you revoke it by deleting a line. No third party is involved at any point, which mattered because the firm’s largest client contract named the parties permitted to process its data. The cost was the eight people it would eventually have to cover, most of them on Windows laptops, and a support burden that would land on the one developer who understood it.

A managed WireGuard mesh. Each device gets a stable private address, nothing is published, and a lost laptop is removed centrally from a console. Considerably easier for eight non-specialists. The honest caveat is that a managed mesh means a company coordinates the key exchange, and although it never sees the traffic, the partners had to decide whether a contract naming permitted processors covered a control plane that handles keys and not data.

A self-hosted control server was the third possibility, which removes the third party and adds a service the firm has to run and keep patched.

The company had bought the card eighteen months earlier to replace a hosted API bill, and by most accounts it had worked. The bill had gone. What had also happened, without anyone writing it down, was that the content team had grown, the chat assistant had become the thing people complained about at eleven in the morning, and somebody had written a proposal for a second card.

The finance director asked for the cost per million tokens before approving it. Nobody had one.

The number that mattered turned out to be the denominator. The gateway had been counting tokens for months for its own rate limiting, so the total served in a month was available immediately: about ninety-four million. A saturating load test, run at a concurrency high enough that the queue never emptied, put the machine’s sustained peak at roughly 1,150 tokens a second. That is about 2,980 million tokens a month at full duty. Ninety-four against 2,980 is a duty cycle of just over three per cent.

The team’s own estimate, collected before the measurement out of curiosity, had averaged around thirty per cent. That gap is normal. Thinking, reading and typing take far longer than generation does, and a developer who feels they use the model constantly is typically under five per cent.

What three per cent does to the arithmetic

The fixed costs run whether or not anything is being generated. Amortisation runs on the calendar. The machine drew about ninety watts idle for the other ninety-seven per cent of the month, which at the local tariff was more than the electricity it used while working. Add the capital cost over a thirty-six-month window and two hours a month of somebody’s attention, and the all-in figure came to a little over a hundred and sixty a month for ninety-four million tokens: about 1.70 per million.

The hosted price for the open model they were running was well under half that. The card was not saving money at the volume it was actually doing. It had saved money in the first three months, when a one-off backfill of two years of archived question banks had run it flat out for a fortnight.

The decision

So the proposal for a second card was the wrong question, and the right one had a cost on both sides.

Buy the second card. It solves the eleven-o’clock complaint directly and keeps everything inside the building. It also doubles the fixed cost of a machine already idle ninety-seven per cent of the time, and it requires capital approval, which in this company took about three months against a monthly subscription that took an afternoon.

Move the chat assistant to a hosted API and keep the card saturated on the nightly batch. Cheaper on these numbers, immediate, and it raises the card’s duty cycle by removing the interactive work it was bad at and leaving the batch work it was good at. The cost is the reason the card was bought: the chat assistant handles unpublished question banks, which are the company’s actual product, and sending them to a third party is a decision about the business rather than about tokens.

There was a third option with no capital in it at all, and it was the one the course points at. The duty cycle is a denominator, and there was work sitting on the other side of the building that could raise it: re-tagging the archive, generating distractor options, backfilling embeddings over every paper the company owned. None of it was urgent and all of it was real.

The desk received about four hundred messages a day across a web form, email and a messaging app. A triage step assigned each one an intent and an urgency, and a general model with a long prompt had been doing it adequately: the prompt ran to about two thousand tokens of category definitions, and it was sent on every single request.

The case for training was one the course recognises as sound. The behaviour existed and was demonstrable, the categories had subtle boundaries that several pages of definitions had failed to express, and folding the behaviour into the weights would remove two thousand tokens of prefill from four hundred requests a day. They wrote the sentence first, as instructed: after this, the model will assign intent and urgency, measured by exact match on a held-out set, on messages of the shape we actually receive.

The dataset took most of three weeks. Eighteen hundred real messages with the correct labels, rendered with the model’s own chat template and read before training, loss computed on the assistant tokens only, ten per cent held out and untouched, a further two hundred collected a month later from a different source so that the test set did not share the collection’s quirks. About a tenth of the examples were messages the desk should decline to triage automatically, because a dataset of successes teaches a model to always succeed.

The run itself was under an hour. Rank 16, alpha 32, two epochs, an effective batch of sixteen, the MLP projections included as well as attention. Evaluation loss fell with training loss and turned upward at step 150, so they took that checkpoint rather than the final one. Exact match on the held-out set was 0.91 against 0.78 for the prompted base.

The check somebody nearly skipped

The twenty-item general-capability set had been built before the run, because the course is insistent about it: neither loss curve detects catastrophic forgetting, and an evaluation set drawn from the same narrow distribution as the training set cannot either. It contained three general knowledge questions, three unrelated instruction-following items, two items in each language the desk’s customers actually use, two that should be refused, and two short reasoning items.

Run at temperature 0 against the base and against the adapter, side by side, it took twenty minutes. The task items were better. The Hindi items were markedly worse: the adapter answered them in English, and where it did answer in Hindi it did so badly. About a third of the desk’s incoming messages are in Hindi or in transliterated Hinglish, and the training set, assembled from the ticketing system’s English export because that was the export that existed, contained almost none.

The decision, under a deadline

The quarter ended in nine days and the triage improvement had been promised.

Ship it with a language router. Detect the language, send Hindi and Hinglish messages to the base model with the old two-thousand-token prompt, and send everything else to the adapter. It works, it ships on time, and it keeps most of the gain. The cost is two behaviours to maintain, two prompts to keep in step, inconsistent labelling between the two paths, and a quiet inequality in which a third of customers get the worse system.

Rebuild the dataset and retrain. Ten days of collecting and labelling Hindi and Hinglish examples, then another run and another evaluation. It misses the quarter and it produces one model that serves everybody the same way.