Running Models Yourself

A practical course on running large language models on your own hardware. It teaches the arithmetic that decides whether a model fits, what quantisation actually costs you, which serving engine wins under which conditions, and when local is genuinely cheaper than an API — with real commands, real numbers, and no engine advertising. Written for people on ordinary laptops as much as for people with a 4090.

1. 1Why run a model yourself, and when not to6 minThe weights set the quality ceiling; hardware only decides whether you can reach it and how fast.
2. 2Does it fit: the arithmetic9 minWeights are fixed; the KV cache grows with context and layers, and that is what usually breaks the budget.
3. 3Quantisation, properly9 minQuantisation damages precision-critical work first — syntax, long instructions, rare tokens — while fluency stays intact.
4. 4Ollama and llama.cpp8 minOllama is llama.cpp plus a registry, a template and sane defaults — leave it only for a flag it does not expose.
5. 5What actually runs on 8GB8 minGeneration speed is memory bandwidth divided by model size; cores and capacity do not change that ceiling.
6. 6Serving engines, and what "faster" means10 minChoose an engine by your concurrency pattern; single-stream speed and aggregate throughput are different, opposing numbers.
7. 7Sampling: why output quality is often not the model8 minThe sampler chooses; before blaming the model, check the template, the context, and one knob at a time.
8. 8Interfaces: what each one is for6 minThe interface never changes the model, only what gets sent — so compare requests, not apps.
9. 9Serving it to other people9 minBind to localhost and reach it over a private network; an open inference port is free compute for strangers.
10. 10The honest cost comparison10 minLocal cost per token is set by how busy the GPU is; an idle card generates the most expensive tokens there are.