Working out what your machine will run

The arithmetic, including the part people forget

The weights are the obvious cost. At 4-bit quantisation a model needs roughly half a byte per parameter, so an 8-billion-parameter model is about 4.5 GB on disk and a similar amount in memory.

The part that surprises people is the KV cache: the model’s working memory for the conversation so far. It grows linearly with how much text is in the context, and on a long document it can rival the weights.

8B model, 4-bit weights                  ~4.5 GB
KV cache at   4,000 tokens of context    ~0.5 GB
KV cache at  32,000 tokens of context    ~4.0 GB
plus the runtime and the operating system ~2.0 GB
                                          -------
short chats:      ~7 GB       long documents: ~10.5 GB

So a machine that runs an 8B model beautifully for chat may swap to disk and crawl the moment you paste a fifty-page report in. That is the single most common “it worked yesterday” complaint, and the fix is not a bigger model: it is a smaller context setting, or 8-bit cache quantisation, which most runners now offer and which roughly halves that column.

What your hardware actually gives you

Ordinary laptop, 8 GB RAM. 3B to 4B models at 4-bit, short contexts. Useful for drafting, rewriting and summarising pasted text.

16 GB. 7B to 9B comfortably, or a 12B at a squeeze. This is where a local model becomes a real daily tool.

32 GB. 24B to 32B at 4-bit. Quality here is good for everyday work.

Apple Silicon. Unified memory means the GPU can use most of the system RAM, which is why a Mac with 32 or 64 GB runs models that would need an expensive graphics card on a PC. Roughly 70% of total RAM is available to the model by default.

A discrete GPU. What matters is VRAM, not the card’s name. 8 GB fits a 7B comfortably; 12 GB fits a 14B; 24 GB fits a 32B, or a 70B at low quantisation with reduced context. Anything that does not fit in VRAM spills into system RAM and slows down by an order of magnitude: not a little, an order of magnitude.

Measuring rather than guessing

Every runner reports speed. Look for tokens per second after a real prompt, not a one-word one.

ollama run qwen3:8b --verbose        # prints eval rate at the end

The interpretation:

• Under 5 tokens/sec: painful to read along with. Fine for a job you leave running.
• 10 to 20: comfortable conversation, roughly reading pace.
• 30 and above: feels instant.

Watch the second number too, prompt evaluation speed. That is how long it takes to read your input before answering. On a CPU it can be slow enough that a fifty-page document takes a minute before the first word appears, even though generation afterwards is fine.

When it is too slow

In order of what to try, cheapest first:

1. Reduce the context setting. Most runners default to something large. If your work is short questions, 4,000 tokens is plenty and it frees a lot of memory.
2. Quantise the KV cache to 8-bit. A setting in llama.cpp, Ollama and LM Studio. Roughly halves cache memory for a negligible quality cost.
3. Drop one quantisation level, Q5 to Q4. Small quality cost, real speed gain.
4. Use a smaller model. A 4B that runs at 25 tokens/sec is more useful than an 8B that runs at 3.
5. Offload some layers to the GPU if you have one. Even a modest card holding half the layers helps considerably.
6. Close the browser. Not a joke: browsers routinely hold several gigabytes, and reclaiming them is often the difference between fitting and swapping.

The honest ceiling

There is a size above which your machine cannot go, and no setting changes it. If a model does not fit, it will either refuse to load or run at a speed that makes it unusable. That is not a failure of configuration.

At that point the choices are a smaller model, a hosted open model at a few cents an hour, or accepting that this particular task is not one your hardware does. All three are reasonable. Buying a graphics card to save a subscription is arithmetic worth doing carefully before, not after: the card costs more than several years of the subscription, and the models it fits will keep changing.