{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"a-model-on-your-own-machine","title":"Running a model on a laptop with no GPU","url":"https://addaly.com/learn/building-with-ai/a-model-on-your-own-machine","position":9,"of":80,"minutes":10,"answer":"A quantised 7B model is about 4 GB, runs on an ordinary laptop through an OpenAI-compatible endpoint on localhost, and is close to hosted quality on classification.","keyIdea":"A quantised 7B model is about 4 GB, runs on an ordinary laptop through an OpenAI-compatible endpoint on localhost, and is close to hosted quality on classification, extraction and rewriting while clearly behind on multi-step reasoning."},"body":{"format":"markdown","text":"## Why this is worth an afternoon\n\nThere are four reasons to run a model locally, and only one of them is cost.\n\n**Nothing leaves the machine.** For medical notes, legal documents, salary data, or anything a client has told you not to send anywhere, local inference is the difference between a project you can take and one you cannot.\n\n**No per-call bill.** You can run ten thousand experimental prompts overnight while you sleep. That changes how you learn, because the meter is no longer a reason to stop trying things.\n\n**No account, no card, no region block.** A student in a town with unreliable payments can do everything in this course on a second-hand laptop.\n\n**You see the machinery.** Context length, quantisation, KV cache, tokens per second stop being words on a pricing page and become numbers you watch on your own screen.\n\n## Quantisation, in one paragraph\n\nA model's weights are trained in 16-bit floating point. A 7-billion-parameter model in 16-bit is about 14 GB, which will not fit in ordinary RAM alongside an operating system. **Quantisation** stores each weight in fewer bits (commonly 4) using a shared scale factor per small block of weights. At 4 bits, that 7B model is about 4.1 GB. Quality loss at 4-bit is small and measurable: perplexity rises slightly, and it shows up first on long multi-step reasoning, rarely on summarising or classification. Below 3 bits the degradation becomes obvious.\n\nThe naming you will see is `Q4_K_M` and friends: 4 bits, K-quant scheme, medium variant. `Q4_K_M` is the default worth starting from. `Q8_0` is nearly lossless and twice the size. `Q2_K` is a curiosity.\n\nRough sizing at Q4:\n\n- **1 to 3B parameters**: 1 to 2 GB. Runs on almost anything, including a phone. Good for classification and simple rewriting.\n- **7 to 8B**: 4 to 5 GB. The sweet spot on a laptop with 8 to 16 GB of RAM. Genuinely useful for summarising, extraction, drafting.\n- **13 to 14B**: 8 to 9 GB. Needs 16 GB comfortably.\n- **30B+**: 18 GB and up. A workstation question.\n\nOn CPU only, expect roughly 5 to 15 tokens per second for a 7B model: slower than reading speed, so it feels sluggish but is entirely usable for background work. Apple Silicon is unusually good here because the GPU shares memory with the CPU: an M-series machine with 16 GB runs a 7B model at 20 to 40 tokens per second.\n\n## The shortest path that works\n\n```bash\ncurl -fsSL https://ollama.com/install.sh | sh   # macOS and Linux\nollama run qwen2.5:7b\n```\n\nThat downloads a quantised model and drops you into a chat. Two commands, no Python, no GPU, no key.\n\nThe part that matters for building is that Ollama serves an **OpenAI-compatible HTTP API** on `localhost:11434`. Your existing code becomes local by changing two lines:\n\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\nr = client.chat.completions.create(\n    model=\"qwen2.5:7b\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarise this in three bullets:\\n\" + text}],\n    temperature=0)\n```\n\nSame request shape, same response shape, same retry code. This is the practical reason to build against a narrow interface: your development loop can run offline and free, and production can point at a hosted model.\n\nThe alternatives, all free: **llama.cpp** is the engine underneath most of this and gives you every flag, including how many layers to offload to a GPU if you have one. **LM Studio** is a graphical application if you would rather not use a terminal. **GPT4All** is similar. For serving many concurrent users on a real GPU, **vLLM** is the one to know, and it is not a laptop tool.\n\n## What you will actually notice\n\nBe honest with yourself about the gap, because it is task-dependent rather than uniform.\n\nA 7B model at Q4 is close to useful hosted models on: classifying a message into one of eight categories, extracting three named fields from clean text, rewriting a paragraph in simpler language, translating between major languages, drafting boilerplate.\n\nIt is clearly worse on: multi-step reasoning, arithmetic, following a complex JSON schema without drift, using more than two or three tools, and anything needing broad world knowledge. It also has a smaller usable context: nominally 32k or 128k, but memory for the KV cache grows with context length, and a long prompt on a 16 GB laptop will swap and crawl.\n\nOne more practical trap: **local models are far more sensitive to prompt format**. Each family expects its own chat template, and Ollama applies the right one for you. Bypass it, send raw text to a completion endpoint, and quality collapses in a way that looks like the model being stupid rather than the template being wrong.\n\nStart with a 7B at Q4, keep a hosted model for the hard cases, and you have the two ends of the routing pattern from the previous lesson running on the same interface."},"check":{"question":"A developer runs a 7B model locally at Q4 and finds it excellent at tagging support tickets but poor at a five-step planning task. What does this pattern indicate?","options":[{"text":"The quantisation is too aggressive and an 8-bit version would fix the planning task.","correct":false,"tempting":"Overstates what quantisation costs; moving from 4-bit to 8-bit recovers a small amount of quality and would not close a gap of this size."},{"text":"The chat template is wrong for the planning prompts.","correct":false,"tempting":"A real failure mode that would degrade everything, including the tagging that is working well, rather than only the multi-step task."},{"text":"Capability loss from a smaller model is concentrated in multi-step reasoning, while single-shot classification degrades least, which is exactly what routing exploits.","correct":true},{"text":"The context window is too small for planning prompts.","correct":false,"tempting":"Attributes the failure to length when a five-step planning prompt is typically short; the difficulty is in chaining decisions, not in fitting them."}]},"prev":{"slug":"choosing-a-model","title":"Choosing a model, and designing so you can change your mind","url":"https://addaly.com/learn/building-with-ai/choosing-a-model"},"next":{"slug":"images-files-and-audio-in","title":"Sending images, PDFs and audio, and what they cost","url":"https://addaly.com/learn/building-with-ai/images-files-and-audio-in"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Running a model on a laptop with no GPU', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/a-model-on-your-own-machine (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Running a model on a laptop with no GPU","courseTitle":"Building with AI","module":{"index":1,"slug":"the-call-underneath-everything","title":"The call underneath everything"},"answer":null,"keyIdea":"A quantised 7B model is about 4 GB, runs on an ordinary laptop through an OpenAI-compatible endpoint on localhost, and is close to hosted quality on classification, extraction and rewriting while clearly behind on multi-step reasoning.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-this-is-worth-an-afternoon","runs":[{"text":"Why this is worth an afternoon"}]},{"t":"p","runs":[{"text":"There are four reasons to run a model locally, and only one of them is cost."}]},{"t":"p","runs":[{"text":"Nothing leaves the machine.","b":true},{"text":" For medical notes, legal documents, salary data, or anything a client has told you not to send anywhere, local inference is the difference between a project you can take and one you cannot."}]},{"t":"p","runs":[{"text":"No per-call bill.","b":true},{"text":" You can run ten thousand experimental prompts overnight while you sleep. That changes how you learn, because the meter is no longer a reason to stop trying things."}]},{"t":"p","runs":[{"text":"No account, no card, no region block.","b":true},{"text":" A student in a town with unreliable payments can do everything in this course on a second-hand laptop."}]},{"t":"p","runs":[{"text":"You see the machinery.","b":true},{"text":" Context length, quantisation, KV cache, tokens per second stop being words on a pricing page and become numbers you watch on your own screen."}]},{"t":"h2","id":"quantisation-in-one-paragraph","runs":[{"text":"Quantisation, in one paragraph"}]},{"t":"p","runs":[{"text":"A model’s weights are trained in 16-bit floating point. A 7-billion-parameter model in 16-bit is about 14 GB, which will not fit in ordinary RAM alongside an operating system. "},{"text":"Quantisation","b":true},{"text":" stores each weight in fewer bits (commonly 4) using a shared scale factor per small block of weights. At 4 bits, that 7B model is about 4.1 GB. Quality loss at 4-bit is small and measurable: perplexity rises slightly, and it shows up first on long multi-step reasoning, rarely on summarising or classification. Below 3 bits the degradation becomes obvious."}]},{"t":"p","runs":[{"text":"The naming you will see is "},{"text":"Q4_K_M","code":true},{"text":" and friends: 4 bits, K-quant scheme, medium variant. "},{"text":"Q4_K_M","code":true},{"text":" is the default worth starting from. "},{"text":"Q8_0","code":true},{"text":" is nearly lossless and twice the size. "},{"text":"Q2_K","code":true},{"text":" is a curiosity."}]},{"t":"p","runs":[{"text":"Rough sizing at Q4:"}]},{"t":"ul","items":[[{"text":"1 to 3B parameters","b":true},{"text":": 1 to 2 GB. Runs on almost anything, including a phone. Good for classification and simple rewriting."}],[{"text":"7 to 8B","b":true},{"text":": 4 to 5 GB. The sweet spot on a laptop with 8 to 16 GB of RAM. Genuinely useful for summarising, extraction, drafting."}],[{"text":"13 to 14B","b":true},{"text":": 8 to 9 GB. Needs 16 GB comfortably."}],[{"text":"30B+","b":true},{"text":": 18 GB and up. A workstation question."}]]},{"t":"p","runs":[{"text":"On CPU only, expect roughly 5 to 15 tokens per second for a 7B model: slower than reading speed, so it feels sluggish but is entirely usable for background work. Apple Silicon is unusually good here because the GPU shares memory with the CPU: an M-series machine with 16 GB runs a 7B model at 20 to 40 tokens per second."}]},{"t":"h2","id":"the-shortest-path-that-works","runs":[{"text":"The shortest path that works"}]},{"t":"code","lang":"bash","label":"Bash","text":"curl -fsSL https://ollama.com/install.sh | sh   # macOS and Linux\nollama run qwen2.5:7b"},{"t":"p","runs":[{"text":"That downloads a quantised model and drops you into a chat. Two commands, no Python, no GPU, no key."}]},{"t":"p","runs":[{"text":"The part that matters for building is that Ollama serves an "},{"text":"OpenAI-compatible HTTP API","b":true},{"text":" on "},{"text":"localhost:11434","code":true},{"text":". Your existing code becomes local by changing two lines:"}]},{"t":"code","lang":"python","label":"Python","text":"from openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\nr = client.chat.completions.create(\n    model=\"qwen2.5:7b\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarise this in three bullets:\\n\" + text}],\n    temperature=0)"},{"t":"p","runs":[{"text":"Same request shape, same response shape, same retry code. This is the practical reason to build against a narrow interface: your development loop can run offline and free, and production can point at a hosted model."}]},{"t":"p","runs":[{"text":"The alternatives, all free: "},{"text":"llama.cpp","b":true},{"text":" is the engine underneath most of this and gives you every flag, including how many layers to offload to a GPU if you have one. "},{"text":"LM Studio","b":true},{"text":" is a graphical application if you would rather not use a terminal. "},{"text":"GPT4All","b":true},{"text":" is similar. For serving many concurrent users on a real GPU, "},{"text":"vLLM","b":true},{"text":" is the one to know, and it is not a laptop tool."}]},{"t":"h2","id":"what-you-will-actually-notice","runs":[{"text":"What you will actually notice"}]},{"t":"p","runs":[{"text":"Be honest with yourself about the gap, because it is task-dependent rather than uniform."}]},{"t":"p","runs":[{"text":"A 7B model at Q4 is close to useful hosted models on: classifying a message into one of eight categories, extracting three named fields from clean text, rewriting a paragraph in simpler language, translating between major languages, drafting boilerplate."}]},{"t":"p","runs":[{"text":"It is clearly worse on: multi-step reasoning, arithmetic, following a complex JSON schema without drift, using more than two or three tools, and anything needing broad world knowledge. It also has a smaller usable context: nominally 32k or 128k, but memory for the KV cache grows with context length, and a long prompt on a 16 GB laptop will swap and crawl."}]},{"t":"p","runs":[{"text":"One more practical trap: "},{"text":"local models are far more sensitive to prompt format","b":true},{"text":". Each family expects its own chat template, and Ollama applies the right one for you. Bypass it, send raw text to a completion endpoint, and quality collapses in a way that looks like the model being stupid rather than the template being wrong."}]},{"t":"p","runs":[{"text":"Start with a 7B at Q4, keep a hosted model for the hard cases, and you have the two ends of the routing pattern from the previous lesson running on the same interface."}]}],"check":{"question":[{"text":"A developer runs a 7B model locally at Q4 and finds it excellent at tagging support tickets but poor at a five-step planning task. What does this pattern indicate?"}],"options":[{"text":[{"text":"The quantisation is too aggressive and an 8-bit version would fix the planning task."}],"correct":false,"tempting":[{"text":"Overstates what quantisation costs; moving from 4-bit to 8-bit recovers a small amount of quality and would not close a gap of this size."}]},{"text":[{"text":"The chat template is wrong for the planning prompts."}],"correct":false,"tempting":[{"text":"A real failure mode that would degrade everything, including the tagging that is working well, rather than only the multi-step task."}]},{"text":[{"text":"Capability loss from a smaller model is concentrated in multi-step reasoning, while single-shot classification degrades least, which is exactly what routing exploits."}],"correct":true,"tempting":null},{"text":[{"text":"The context window is too small for planning prompts."}],"correct":false,"tempting":[{"text":"Attributes the failure to length when a five-step planning prompt is typically short; the difficulty is in chaining decisions, not in fitting them."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"choosing-a-model","title":"Choosing a model, and designing so you can change your mind","minutes":9},"next":{"slug":"images-files-and-audio-in","title":"Sending images, PDFs and audio, and what they cost","minutes":9}}}