Why run a model yourself, and when not to

Almost everything written about local models is either an advertisement or an install guide. Start with the question those skip: should you do this at all?

What running it yourself actually buys

The data never leaves. Clinical notes, legal discovery, an employer's source code, other people's private messages. No processing agreement, no sub-processor list, no question about training on your inputs. For some work this is not a preference, it is the condition of doing the work at all.

No rate limits, no queue, no account. You can hammer it for six hours. Nobody suspends you at 2am mid-batch.

It works with no network. On a plane, on a ship, in a lab with no outbound access, in a country where the API is blocked or the payment method is not accepted. This last one is not a small case.

Nobody deprecates it. Hosted models get retired with a few months' notice, and the replacement behaves differently. A GGUF file on your disk behaves identically in five years. If you have tuned prompts around a specific model's quirks, that stability is worth real money.

Cost, but only under conditions. Local can be dramatically cheaper or dramatically more expensive than an API depending on one variable. Lesson 10 does that arithmetic properly. Do not assume it yet.

The honest reasons not to

The quality gap is real and it is large. An 8B model at 4-bit is a capable 2023-class assistant. It summarises, classifies, extracts, rewrites, and writes short functions well. It does not do long multi-step reasoning, does not hold a large codebase in its head, and calls tools less reliably. No amount of hardware changes this — the weights set the ceiling.

Your time is the largest line item. An afternoon spent debugging a chat template costs more than a year of light API use at any professional rate.

Speed, especially before the first token. On a CPU, generation might be tolerable while processing a 2,000-token prompt takes 30–60 seconds. That delay, not tokens per second, is what makes people give up.

You are the operations team. When it stops working at 3am, that is your problem.

A test that decides it in an hour

Do not start by buying hardware. Separate two questions that people fuse together: *is this model good enough for my task?* and *can I run it?*

Take twenty real prompts from your actual work. Run them against a hosted copy of the exact open weights you would run locally — most open models are available through several providers for a few cents. Grade the outputs yourself.

If the quality is unacceptable there, it will be unacceptable locally, and a bigger GPU will not fix it. If it is acceptable, you now know the target and can shop for the cheapest way to run those specific weights.

This costs about a dollar and it is the single most useful thing in this course.

What you will need to be able to do

The rest of these lessons assume you want to answer four questions without asking anyone:

1. 1Given any model on Hugging Face, does it fit in my memory, and at what context length?
2. 2What did quantisation cost me, and where will I notice?
3. 3Which serving engine wins for my pattern of use, and by what measure of "wins"?
4. 4Is this cheaper than the API for my volume?

Those are arithmetic and engineering questions with real answers. Almost nothing else in this field is.