{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"running-on-your-own-machine","title":"Running a model on your own machine","url":"https://addaly.com/learn/how-llms-work/running-on-your-own-machine","position":62,"of":78,"minutes":9,"answer":"A local model is chosen from the memory you have at 0.6 gigabytes per billion parameters, runs at the bandwidth division.","keyIdea":"A local model is chosen from the memory you have at 0.6 gigabytes per billion parameters, runs at the bandwidth division, and is undone most often by a model that spills into swap or a chat template that does not match the file."},"body":{"format":"markdown","text":"## Why you would, and why you might not\n\nRunning a model on your own hardware buys four things. The text never leaves the machine, which for medical notes, legal drafts or a client's data is not a preference but a requirement. Nothing drifts: module six's version problem disappears, because the file on disk is the model. There is no per-token bill, so an experiment that would cost money against an API costs electricity. And you learn more about how these systems behave from one evening with a local model than from a month with a chat box.\n\nIt costs three things. The quality ceiling is whatever fits in your memory, and that is a long way below the largest served models. Speed follows the bandwidth arithmetic from two lessons ago and is rarely quick on a laptop. And you become the person who maintains it. Local is a real option, not a moral position; take it when the four benefits outweigh the three costs.\n\n## Choose the model from the memory you have\n\nThe previous two lessons give the rule. At four-bit quantisation a model needs about 0.6 gigabytes per billion parameters, plus cache, plus whatever the operating system keeps. Working back from typical machines:\n\n- **8 GB** (many phones, older laptops): 1- to 4-billion-parameter models.\n- **16 GB**: 7- to 8-billion models with a modest context. This is the most common local setup and it is useful.\n- **32 GB**: 14-billion models, or an 8-billion with a long context and room to spare.\n- **64 GB or more**: 32-billion models comfortably; 70-billion models at four bits, slowly.\n\nA model that does not fit does not merely run slowly: it spills into swap and the speed collapses to a token every second or two. If you see that, the fix is a smaller model or a lower bit width, not patience.\n\n## The tools, all free\n\n`llama.cpp` is the engine underneath almost everything local: a C++ program that reads GGUF files, runs on CPUs and every common GPU, and offers a command line and an OpenAI-compatible local server. Ollama wraps it with a model library and one-line commands. LM Studio and Jan add a graphical interface, Jan being open source. On phones, PocketPal and MLC Chat run the same kinds of files. In a browser, WebLLM runs models on the GPU through WebGPU with nothing installed.\n\n```bash\n# Ollama: pulls a four-bit build and starts a chat\nollama run llama3.1:8b\n\n# llama.cpp directly, with a file you chose, all layers on the GPU,\n# an 8,192-token cache, and a local API on port 8080\nllama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 -c 8192 --port 8080\n```\n\nTwo flags do most of the work. `-ngl` is the number of layers to place on the GPU; 99 means all of them, and a smaller number offloads the rest to system memory at the speed penalty described in the fitting lesson. `-c` is the context length, which sets the cache budget: a larger number reserves more memory before you have typed anything.\n\n## What to expect\n\nGeneration speed is the bandwidth division. An 8-billion model at four bits on a laptop with 200 gigabytes a second of memory bandwidth gives 25 to 35 tokens a second, which is faster than most people read. A CPU-only desktop gives about ten. A phone gives ten to twenty on a 3-billion model.\n\nWhere laptops suffer is the prompt. Prefill is compute-bound, and laptop compute is a small fraction of a data-centre card's. A 4,000-token prompt can take two to four seconds on Apple silicon and ten to twenty on a CPU before the first output token appears. Long documents are the case where the local experience falls furthest behind the API.\n\nThe chat template travels inside the GGUF file, as module one noted. The single most common cause of a local model producing nonsense, ignoring instructions, or never stopping is a template mismatch: a file built before the template was fixed, or a front-end applying its own. `llama-server --verbose` shows the exact text reaching the model, and module three's five-minute diagnostic applies unchanged.\n\n## Free GPUs when yours is not enough\n\nGoogle Colab's free tier offers a 16-gigabyte T4 for limited sessions. Kaggle gives around thirty hours a week of GPU time on similar cards. Hugging Face Spaces provides free CPU hosting and quota-limited GPU bursts. Any of these runs a 7- to 13-billion model at four bits and, as module five said, is enough for QLoRA fine-tuning. Sessions end without warning; save anything you care about to Drive or a model hub as you go.\n\n## The complaint to pre-empt\n\n\"It is dumber than the API.\" Usually true, and usually a comparison between a four-bit 8-billion model and a served model fifty times its size. Set the expectation from the parameter count, then be pleasantly surprised: a current 8-billion model is roughly as capable as the best served models were two years earlier, which is a great deal of capability for a laptop.\n\nThree other things go wrong often enough to list. A quantisation too aggressive for a small model (module seven's third lesson) produces a model that is subtly wrong everywhere. A wrong stop token produces a model that answers and then keeps going. And a graphical front-end may send telemetry or check for updates; `llama.cpp` itself contacts nothing, which matters if the reason you went local was privacy.\n\n## When it is the wrong choice\n\nYou need frontier quality, or reliable tool use at the level only the largest models manage, or you need to serve many users at once (the next lesson) and do not want to run the infrastructure. Then the API is the right tool, and knowing what a local model can do is what lets you make that call on evidence rather than on habit."},"check":{"question":"A learner's 16-gigabyte laptop runs an 8-billion-parameter four-bit model at 25 tokens a second. They load a 14-billion four-bit model and get 1.5 tokens a second. What happened, and what is the fix?","options":[{"text":"The larger model needs more CPU threads than the default allocates, so raising the thread count will restore the speed.","correct":false,"tempting":"Threads matter a little for CPU inference, but decode is bandwidth-bound and no thread setting explains a seventeen-fold collapse."},{"text":"The weights plus cache no longer fit in memory and spilled to swap; use a smaller model or a lower bit width.","correct":true},{"text":"A model with 1.75 times the parameters is proportionally slower, so the result is expected and the fix is to accept the speed.","correct":false,"tempting":"The bandwidth division does predict a slowdown, but of about 1.75 times, to roughly 14 tokens a second; the extra order of magnitude is the signature of swapping."},{"text":"The larger model's file carries a different chat template, so it is generating long hidden reasoning tokens before each visible token, which makes the visible rate look far lower than the true rate.","correct":false,"tempting":"Template problems are the commonest local failure, but they produce wrong or endless output, not a slow stream of correct tokens; the rate itself is the clue."}]},"prev":{"slug":"quantisation-mechanics","title":"Quantisation: what the numbers lose","url":"https://addaly.com/learn/how-llms-work/quantisation-mechanics"},"next":{"slug":"serving-many-users","title":"Serving a hundred people from one card","url":"https://addaly.com/learn/how-llms-work/serving-many-users"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Running a model on your own machine', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/running-on-your-own-machine (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Running a model on your own machine","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"A local model is chosen from the memory you have at 0.6 gigabytes per billion parameters, runs at the bandwidth division, and is undone most often by a model that spills into swap or a chat template that does not match the file.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-you-would-and-why-you-might-not","runs":[{"text":"Why you would, and why you might not"}]},{"t":"p","runs":[{"text":"Running a model on your own hardware buys four things. The text never leaves the machine, which for medical notes, legal drafts or a client’s data is not a preference but a requirement. Nothing drifts: module six’s version problem disappears, because the file on disk is the model. There is no per-token bill, so an experiment that would cost money against an API costs electricity. And you learn more about how these systems behave from one evening with a local model than from a month with a chat box."}]},{"t":"p","runs":[{"text":"It costs three things. The quality ceiling is whatever fits in your memory, and that is a long way below the largest served models. Speed follows the bandwidth arithmetic from two lessons ago and is rarely quick on a laptop. And you become the person who maintains it. Local is a real option, not a moral position; take it when the four benefits outweigh the three costs."}]},{"t":"h2","id":"choose-the-model-from-the-memory-you-have","runs":[{"text":"Choose the model from the memory you have"}]},{"t":"p","runs":[{"text":"The previous two lessons give the rule. At four-bit quantisation a model needs about 0.6 gigabytes per billion parameters, plus cache, plus whatever the operating system keeps. Working back from typical machines:"}]},{"t":"ul","items":[[{"text":"8 GB","b":true},{"text":" (many phones, older laptops): 1- to 4-billion-parameter models."}],[{"text":"16 GB","b":true},{"text":": 7- to 8-billion models with a modest context. This is the most common local setup and it is useful."}],[{"text":"32 GB","b":true},{"text":": 14-billion models, or an 8-billion with a long context and room to spare."}],[{"text":"64 GB or more","b":true},{"text":": 32-billion models comfortably; 70-billion models at four bits, slowly."}]]},{"t":"p","runs":[{"text":"A model that does not fit does not merely run slowly: it spills into swap and the speed collapses to a token every second or two. If you see that, the fix is a smaller model or a lower bit width, not patience."}]},{"t":"h2","id":"the-tools-all-free","runs":[{"text":"The tools, all free"}]},{"t":"p","runs":[{"text":"llama.cpp","code":true},{"text":" is the engine underneath almost everything local: a C++ program that reads GGUF files, runs on CPUs and every common GPU, and offers a command line and an OpenAI-compatible local server. Ollama wraps it with a model library and one-line commands. LM Studio and Jan add a graphical interface, Jan being open source. On phones, PocketPal and MLC Chat run the same kinds of files. In a browser, WebLLM runs models on the GPU through WebGPU with nothing installed."}]},{"t":"code","lang":"bash","label":"Bash","text":"# Ollama: pulls a four-bit build and starts a chat\nollama run llama3.1:8b\n\n# llama.cpp directly, with a file you chose, all layers on the GPU,\n# an 8,192-token cache, and a local API on port 8080\nllama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 -c 8192 --port 8080"},{"t":"p","runs":[{"text":"Two flags do most of the work. "},{"text":"-ngl","code":true},{"text":" is the number of layers to place on the GPU; 99 means all of them, and a smaller number offloads the rest to system memory at the speed penalty described in the fitting lesson. "},{"text":"-c","code":true},{"text":" is the context length, which sets the cache budget: a larger number reserves more memory before you have typed anything."}]},{"t":"h2","id":"what-to-expect","runs":[{"text":"What to expect"}]},{"t":"p","runs":[{"text":"Generation speed is the bandwidth division. An 8-billion model at four bits on a laptop with 200 gigabytes a second of memory bandwidth gives 25 to 35 tokens a second, which is faster than most people read. A CPU-only desktop gives about ten. A phone gives ten to twenty on a 3-billion model."}]},{"t":"p","runs":[{"text":"Where laptops suffer is the prompt. Prefill is compute-bound, and laptop compute is a small fraction of a data-centre card’s. A 4,000-token prompt can take two to four seconds on Apple silicon and ten to twenty on a CPU before the first output token appears. Long documents are the case where the local experience falls furthest behind the API."}]},{"t":"p","runs":[{"text":"The chat template travels inside the GGUF file, as module one noted. The single most common cause of a local model producing nonsense, ignoring instructions, or never stopping is a template mismatch: a file built before the template was fixed, or a front-end applying its own. "},{"text":"llama-server --verbose","code":true},{"text":" shows the exact text reaching the model, and module three’s five-minute diagnostic applies unchanged."}]},{"t":"h2","id":"free-gpus-when-yours-is-not-enough","runs":[{"text":"Free GPUs when yours is not enough"}]},{"t":"p","runs":[{"text":"Google Colab’s free tier offers a 16-gigabyte T4 for limited sessions. Kaggle gives around thirty hours a week of GPU time on similar cards. Hugging Face Spaces provides free CPU hosting and quota-limited GPU bursts. Any of these runs a 7- to 13-billion model at four bits and, as module five said, is enough for QLoRA fine-tuning. Sessions end without warning; save anything you care about to Drive or a model hub as you go."}]},{"t":"h2","id":"the-complaint-to-pre-empt","runs":[{"text":"The complaint to pre-empt"}]},{"t":"p","runs":[{"text":"“It is dumber than the API.” Usually true, and usually a comparison between a four-bit 8-billion model and a served model fifty times its size. Set the expectation from the parameter count, then be pleasantly surprised: a current 8-billion model is roughly as capable as the best served models were two years earlier, which is a great deal of capability for a laptop."}]},{"t":"p","runs":[{"text":"Three other things go wrong often enough to list. A quantisation too aggressive for a small model (module seven’s third lesson) produces a model that is subtly wrong everywhere. A wrong stop token produces a model that answers and then keeps going. And a graphical front-end may send telemetry or check for updates; "},{"text":"llama.cpp","code":true},{"text":" itself contacts nothing, which matters if the reason you went local was privacy."}]},{"t":"h2","id":"when-it-is-the-wrong-choice","runs":[{"text":"When it is the wrong choice"}]},{"t":"p","runs":[{"text":"You need frontier quality, or reliable tool use at the level only the largest models manage, or you need to serve many users at once (the next lesson) and do not want to run the infrastructure. Then the API is the right tool, and knowing what a local model can do is what lets you make that call on evidence rather than on habit."}]}],"check":{"question":[{"text":"A learner’s 16-gigabyte laptop runs an 8-billion-parameter four-bit model at 25 tokens a second. They load a 14-billion four-bit model and get 1.5 tokens a second. What happened, and what is the fix?"}],"options":[{"text":[{"text":"The larger model needs more CPU threads than the default allocates, so raising the thread count will restore the speed."}],"correct":false,"tempting":[{"text":"Threads matter a little for CPU inference, but decode is bandwidth-bound and no thread setting explains a seventeen-fold collapse."}]},{"text":[{"text":"The weights plus cache no longer fit in memory and spilled to swap; use a smaller model or a lower bit width."}],"correct":true,"tempting":null},{"text":[{"text":"A model with 1.75 times the parameters is proportionally slower, so the result is expected and the fix is to accept the speed."}],"correct":false,"tempting":[{"text":"The bandwidth division does predict a slowdown, but of about 1.75 times, to roughly 14 tokens a second; the extra order of magnitude is the signature of swapping."}]},{"text":[{"text":"The larger model’s file carries a different chat template, so it is generating long hidden reasoning tokens before each visible token, which makes the visible rate look far lower than the true rate."}],"correct":false,"tempting":[{"text":"Template problems are the commonest local failure, but they produce wrong or endless output, not a slow stream of correct tokens; the rate itself is the clue."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"quantisation-mechanics","title":"Quantisation: what the numbers lose","minutes":10},"next":{"slug":"serving-many-users","title":"Serving a hundred people from one card","minutes":9}}}