It is genuinely one command now
Install Ollama or LM Studio — both free, both work on Windows, macOS and Linux. Then, in a terminal:
ollama run qwen3:8bIt downloads a few gigabytes and gives you a prompt. That is the whole setup. LM Studio does the same with a graphical interface and a model browser, which is friendlier if terminals are not your habit. Underneath both sits llama.cpp, the project that made running these models on ordinary hardware practical.
Quantization, in one paragraph
Model weights are normally stored at 16 bits per number. Quantization stores them at 8, 5 or 4 bits instead. The file shrinks roughly in proportion, the model gets faster, and it loses a little accuracy — at 4 bits, usually a little enough that you will not notice on everyday tasks.
The useful arithmetic: at 4 bits, a model takes roughly half a byte per parameter. So an 8-billion-parameter model is about 5 GB. A 32B is about 20 GB. A 70B is about 40 GB. You need memory somewhat larger than the file, because the conversation itself takes space too.
What your machine can actually run
8 GB of RAM. Models of 3 to 4 billion parameters at 4 bits. Genuinely useful for summarizing, drafting, rewriting, and offline questions. Not for hard reasoning or code you will ship.
16 GB. 7 to 9 billion parameters, comfortably. This is where a local model becomes a daily tool rather than a demo.
32 GB, or an Apple Silicon machine with unified memory. 27 to 32 billion parameters. This is where quality gets genuinely good, and where a lot of people stop needing a subscription for routine work.
64 GB or a dedicated GPU with 24 GB. 70 billion parameters, slowly on CPU, fast on GPU.
Bigger than that means a workstation or rented cloud hardware, and at that point renting a hosted open model is usually cheaper.
Speed matters more than you expect
The number to watch is tokens per second. Below about 5, reading the answer appear is genuinely irritating. Around 15 to 20 it feels like a normal chat. Above 30 it feels instant.
On a modern laptop CPU with no GPU, a 7B model at 4 bits typically manages 5 to 10 tokens per second. On a recent Apple Silicon machine, considerably more. On a gaming GPU, several times more again. Try before you decide it is unusable, and try before you decide it is fine.
Why you might
Confidentiality that is structural. A clinic in Nairobi with patient notes. A lawyer under privilege. An accountant with client books. Nothing is sent anywhere, so no policy needs to be trusted and no breach elsewhere can expose you.
Bad or expensive internet. The model works on a train, on a plane, during an outage, and on a metered connection.
Volume. Once it is on your disk, running it 10,000 times costs electricity. If you need to classify every email in a five-year archive, this is the answer.
Learning. Running one locally makes the machinery visible — the system prompt, the context window filling up, the temperature setting, what happens when you run out of room. People who have done this reason about AI better.
Permanence. No deprecation notice will ever arrive.
Why you might not
The model is usually a year or so behind the frontier on hard tasks. It drains your battery and heats your laptop. It has no web access unless you add it, no memory unless you add it, and no polished mobile app. Setup takes an evening the first time.
For most people this is a second tool, not a replacement — the one you reach for when the material should not leave the building, or when you are offline, or when there are ten thousand of something. Which is the theme of the last lesson: not one tool, but a small set.
Before you move on