Running a model on a laptop with no GPU

Why this is worth an afternoon

There are four reasons to run a model locally, and only one of them is cost.

Nothing leaves the machine. For medical notes, legal documents, salary data, or anything a client has told you not to send anywhere, local inference is the difference between a project you can take and one you cannot.

No per-call bill. You can run ten thousand experimental prompts overnight while you sleep. That changes how you learn, because the meter is no longer a reason to stop trying things.

No account, no card, no region block. A student in a town with unreliable payments can do everything in this course on a second-hand laptop.

You see the machinery. Context length, quantisation, KV cache, tokens per second stop being words on a pricing page and become numbers you watch on your own screen.

Quantisation, in one paragraph

A model’s weights are trained in 16-bit floating point. A 7-billion-parameter model in 16-bit is about 14 GB, which will not fit in ordinary RAM alongside an operating system. Quantisation stores each weight in fewer bits (commonly 4) using a shared scale factor per small block of weights. At 4 bits, that 7B model is about 4.1 GB. Quality loss at 4-bit is small and measurable: perplexity rises slightly, and it shows up first on long multi-step reasoning, rarely on summarising or classification. Below 3 bits the degradation becomes obvious.

The naming you will see is Q4_K_M and friends: 4 bits, K-quant scheme, medium variant. Q4_K_M is the default worth starting from. Q8_0 is nearly lossless and twice the size. Q2_K is a curiosity.

Rough sizing at Q4:

• 1 to 3B parameters: 1 to 2 GB. Runs on almost anything, including a phone. Good for classification and simple rewriting.
• 7 to 8B: 4 to 5 GB. The sweet spot on a laptop with 8 to 16 GB of RAM. Genuinely useful for summarising, extraction, drafting.
• 13 to 14B: 8 to 9 GB. Needs 16 GB comfortably.
• 30B+: 18 GB and up. A workstation question.

On CPU only, expect roughly 5 to 15 tokens per second for a 7B model: slower than reading speed, so it feels sluggish but is entirely usable for background work. Apple Silicon is unusually good here because the GPU shares memory with the CPU: an M-series machine with 16 GB runs a 7B model at 20 to 40 tokens per second.

The shortest path that works

curl -fsSL https://ollama.com/install.sh | sh   # macOS and Linux
ollama run qwen2.5:7b

That downloads a quantised model and drops you into a chat. Two commands, no Python, no GPU, no key.

The part that matters for building is that Ollama serves an OpenAI-compatible HTTP API on localhost:11434. Your existing code becomes local by changing two lines:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Summarise this in three bullets:\n" + text}],
    temperature=0)

Same request shape, same response shape, same retry code. This is the practical reason to build against a narrow interface: your development loop can run offline and free, and production can point at a hosted model.

The alternatives, all free: llama.cpp is the engine underneath most of this and gives you every flag, including how many layers to offload to a GPU if you have one. LM Studio is a graphical application if you would rather not use a terminal. GPT4All is similar. For serving many concurrent users on a real GPU, vLLM is the one to know, and it is not a laptop tool.

What you will actually notice

Be honest with yourself about the gap, because it is task-dependent rather than uniform.

A 7B model at Q4 is close to useful hosted models on: classifying a message into one of eight categories, extracting three named fields from clean text, rewriting a paragraph in simpler language, translating between major languages, drafting boilerplate.

It is clearly worse on: multi-step reasoning, arithmetic, following a complex JSON schema without drift, using more than two or three tools, and anything needing broad world knowledge. It also has a smaller usable context: nominally 32k or 128k, but memory for the KV cache grows with context length, and a long prompt on a 16 GB laptop will swap and crawl.

One more practical trap: local models are far more sensitive to prompt format. Each family expects its own chat template, and Ollama applies the right one for you. Bypass it, send raw text to a completion endpoint, and quality collapses in a way that looks like the model being stupid rather than the template being wrong.

Start with a 7B at Q4, keep a hosted model for the hard cases, and you have the two ends of the routing pattern from the previous lesson running on the same interface.