{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"on-your-laptop","title":"Running a model on your own machine","url":"https://addaly.com/learn/choosing-your-tools/on-your-laptop","position":40,"of":76,"minutes":9,"answer":"A local model buys privacy, offline access and zero marginal cost, and charges you capability and speed for it.","keyIdea":"A local model buys privacy, offline access and zero marginal cost, and charges you capability and speed for it."},"body":{"format":"markdown","text":"## It is one command now\n\nInstall Ollama or LM Studio: both free, both work on Windows, macOS and Linux. Then, in a terminal:\n\n```bash\nollama run qwen3:8b\n```\n\nIt downloads a few gigabytes and gives you a prompt. That is the whole setup. LM Studio does the same with a graphical interface and a model browser, which is friendlier if terminals are not your habit. Underneath both sits llama.cpp, the project that made running these models on ordinary hardware practical.\n\n## Quantization, in one paragraph\n\nModel weights are normally stored at 16 bits per number. Quantization stores them at 8, 5 or 4 bits instead. The file shrinks roughly in proportion, the model gets faster, and it loses a little accuracy: at 4 bits, usually a little enough that you will not notice on everyday tasks.\n\nThe useful arithmetic: at 4 bits, a model takes roughly half a byte per parameter. So an 8-billion-parameter model is about 5 GB. A 32B is about 20 GB. A 70B is about 40 GB. You need memory somewhat larger than the file, because the conversation itself takes space too.\n\n## What your machine can actually run\n\n**8 GB of RAM.** Models of 3 to 4 billion parameters at 4 bits. Genuinely useful for summarizing, drafting, rewriting, and offline questions. Not for hard reasoning or code you will ship.\n\n**16 GB.** 7 to 9 billion parameters, comfortably. This is where a local model becomes a daily tool rather than a demo.\n\n**32 GB, or an Apple Silicon machine with unified memory.** 27 to 32 billion parameters. This is where quality gets good, and where a lot of people stop needing a subscription for routine work.\n\n**64 GB or a dedicated GPU with 24 GB.** 70 billion parameters, slowly on CPU, fast on GPU.\n\nBigger than that means a workstation or rented cloud hardware, and at that point renting a hosted open model is usually cheaper.\n\n## Speed matters more than you expect\n\nThe number to watch is tokens per second. Below about 5, reading the answer appear is irritating. Around 15 to 20 it feels like a normal chat. Above 30 it feels instant.\n\nOn a modern laptop CPU with no GPU, a 7B model at 4 bits typically manages 5 to 10 tokens per second. On a recent Apple Silicon machine, considerably more. On a gaming GPU, several times more again. Try before you decide it is unusable, and try before you decide it is fine.\n\n## Why you might\n\n**Confidentiality that is structural.** A clinic in Nairobi with patient notes. A lawyer under privilege. An accountant with client books. Nothing is sent anywhere, so no policy needs to be trusted and no breach elsewhere can expose you.\n\n**Bad or expensive internet.** The model works on a train, on a plane, during an outage, and on a metered connection.\n\n**Volume.** Once it is on your disk, running it 10,000 times costs electricity. If you need to classify every email in a five-year archive, this is the answer.\n\n**Learning.** Running one locally makes the machinery visible: the system prompt, the context window filling up, the temperature setting, what happens when you run out of room. People who have done this reason about AI better.\n\n**Permanence.** No deprecation notice will ever arrive.\n\n## Why you might not\n\nThe model is usually a year or so behind the frontier on hard tasks. It drains your battery and heats your laptop. It has no web access unless you add it, no memory unless you add it, and no polished mobile app. Setup takes an evening the first time.\n\nFor most people this is a second tool, not a replacement: the one you reach for when the material should not leave the building, or when you are offline, or when there are ten thousand of something. Which is the theme of the last lesson: not one tool, but a small set."},"check":{"question":"Ana's laptop has 16 GB of RAM. She downloads a 70B model quantized to 4 bits (about 40 GB) and it either refuses to start or runs unbearably slowly. Why?","options":[{"text":"The weights have to sit in memory while running, and 40 GB does not fit in 16 GB, so the machine falls back to disk or gives up.","correct":true},{"text":"4-bit quantization is too aggressive, and 70B models only work correctly at 16-bit precision.","correct":false,"tempting":"4-bit costs a little quality but works fine; the failure here is capacity, not precision."},{"text":"70B models require a dedicated GPU by design and cannot run on a CPU at all.","correct":false,"tempting":"CPU inference works, just slowly. Model size, not processor type, is what stops her."},{"text":"The download is corrupt, since a correctly quantized 70B model should be around 8 GB.","correct":false,"tempting":"4-bit is roughly half a byte per parameter, so 70B lands near 40 GB: the file size is right."}]},"prev":{"slug":"open-models","title":"The open models","url":"https://addaly.com/learn/choosing-your-tools/open-models"},"next":{"slug":"sizing-for-your-machine","title":"Working out what your machine will run","url":"https://addaly.com/learn/choosing-your-tools/sizing-for-your-machine"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Running a model on your own machine', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/on-your-laptop (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Running a model on your own machine","courseTitle":"Choosing and Using the Tools","module":{"index":5,"slug":"open-models-and-your-own-hardware","title":"Open models and your own hardware"},"answer":null,"keyIdea":"A local model buys privacy, offline access and zero marginal cost, and charges you capability and speed for it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"it-is-one-command-now","runs":[{"text":"It is one command now"}]},{"t":"p","runs":[{"text":"Install Ollama or LM Studio: both free, both work on Windows, macOS and Linux. Then, in a terminal:"}]},{"t":"code","lang":"bash","label":"Bash","text":"ollama run qwen3:8b"},{"t":"p","runs":[{"text":"It downloads a few gigabytes and gives you a prompt. That is the whole setup. LM Studio does the same with a graphical interface and a model browser, which is friendlier if terminals are not your habit. Underneath both sits llama.cpp, the project that made running these models on ordinary hardware practical."}]},{"t":"h2","id":"quantization-in-one-paragraph","runs":[{"text":"Quantization, in one paragraph"}]},{"t":"p","runs":[{"text":"Model weights are normally stored at 16 bits per number. Quantization stores them at 8, 5 or 4 bits instead. The file shrinks roughly in proportion, the model gets faster, and it loses a little accuracy: at 4 bits, usually a little enough that you will not notice on everyday tasks."}]},{"t":"p","runs":[{"text":"The useful arithmetic: at 4 bits, a model takes roughly half a byte per parameter. So an 8-billion-parameter model is about 5 GB. A 32B is about 20 GB. A 70B is about 40 GB. You need memory somewhat larger than the file, because the conversation itself takes space too."}]},{"t":"h2","id":"what-your-machine-can-actually-run","runs":[{"text":"What your machine can actually run"}]},{"t":"p","runs":[{"text":"8 GB of RAM.","b":true},{"text":" Models of 3 to 4 billion parameters at 4 bits. Genuinely useful for summarizing, drafting, rewriting, and offline questions. Not for hard reasoning or code you will ship."}]},{"t":"p","runs":[{"text":"16 GB.","b":true},{"text":" 7 to 9 billion parameters, comfortably. This is where a local model becomes a daily tool rather than a demo."}]},{"t":"p","runs":[{"text":"32 GB, or an Apple Silicon machine with unified memory.","b":true},{"text":" 27 to 32 billion parameters. This is where quality gets good, and where a lot of people stop needing a subscription for routine work."}]},{"t":"p","runs":[{"text":"64 GB or a dedicated GPU with 24 GB.","b":true},{"text":" 70 billion parameters, slowly on CPU, fast on GPU."}]},{"t":"p","runs":[{"text":"Bigger than that means a workstation or rented cloud hardware, and at that point renting a hosted open model is usually cheaper."}]},{"t":"h2","id":"speed-matters-more-than-you-expect","runs":[{"text":"Speed matters more than you expect"}]},{"t":"p","runs":[{"text":"The number to watch is tokens per second. Below about 5, reading the answer appear is irritating. Around 15 to 20 it feels like a normal chat. Above 30 it feels instant."}]},{"t":"p","runs":[{"text":"On a modern laptop CPU with no GPU, a 7B model at 4 bits typically manages 5 to 10 tokens per second. On a recent Apple Silicon machine, considerably more. On a gaming GPU, several times more again. Try before you decide it is unusable, and try before you decide it is fine."}]},{"t":"h2","id":"why-you-might","runs":[{"text":"Why you might"}]},{"t":"p","runs":[{"text":"Confidentiality that is structural.","b":true},{"text":" A clinic in Nairobi with patient notes. A lawyer under privilege. An accountant with client books. Nothing is sent anywhere, so no policy needs to be trusted and no breach elsewhere can expose you."}]},{"t":"p","runs":[{"text":"Bad or expensive internet.","b":true},{"text":" The model works on a train, on a plane, during an outage, and on a metered connection."}]},{"t":"p","runs":[{"text":"Volume.","b":true},{"text":" Once it is on your disk, running it 10,000 times costs electricity. If you need to classify every email in a five-year archive, this is the answer."}]},{"t":"p","runs":[{"text":"Learning.","b":true},{"text":" Running one locally makes the machinery visible: the system prompt, the context window filling up, the temperature setting, what happens when you run out of room. People who have done this reason about AI better."}]},{"t":"p","runs":[{"text":"Permanence.","b":true},{"text":" No deprecation notice will ever arrive."}]},{"t":"h2","id":"why-you-might-not","runs":[{"text":"Why you might not"}]},{"t":"p","runs":[{"text":"The model is usually a year or so behind the frontier on hard tasks. It drains your battery and heats your laptop. It has no web access unless you add it, no memory unless you add it, and no polished mobile app. Setup takes an evening the first time."}]},{"t":"p","runs":[{"text":"For most people this is a second tool, not a replacement: the one you reach for when the material should not leave the building, or when you are offline, or when there are ten thousand of something. Which is the theme of the last lesson: not one tool, but a small set."}]}],"check":{"question":[{"text":"Ana’s laptop has 16 GB of RAM. She downloads a 70B model quantized to 4 bits (about 40 GB) and it either refuses to start or runs unbearably slowly. Why?"}],"options":[{"text":[{"text":"The weights have to sit in memory while running, and 40 GB does not fit in 16 GB, so the machine falls back to disk or gives up."}],"correct":true,"tempting":null},{"text":[{"text":"4-bit quantization is too aggressive, and 70B models only work correctly at 16-bit precision."}],"correct":false,"tempting":[{"text":"4-bit costs a little quality but works fine; the failure here is capacity, not precision."}]},{"text":[{"text":"70B models require a dedicated GPU by design and cannot run on a CPU at all."}],"correct":false,"tempting":[{"text":"CPU inference works, just slowly. Model size, not processor type, is what stops her."}]},{"text":[{"text":"The download is corrupt, since a correctly quantized 70B model should be around 8 GB."}],"correct":false,"tempting":[{"text":"4-bit is roughly half a byte per parameter, so 70B lands near 40 GB: the file size is right."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"open-models","title":"The open models","minutes":8},"next":{"slug":"sizing-for-your-machine","title":"Working out what your machine will run","minutes":9}}}