{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"running-one-on-your-own-machine","title":"A model on your own laptop or phone","url":"https://addaly.com/learn/ai-actually-explained/running-one-on-your-own-machine","position":66,"of":92,"minutes":10,"answer":"Shrinking each of a model's numbers from sixteen bits to four makes an eight-billion-parameter model fit in five gigabytes and run on an ordinary laptop.","keyIdea":"Shrinking each of a model's numbers from sixteen bits to four makes an eight-billion-parameter model fit in five gigabytes and run on an ordinary laptop, where it is slower and thinner on facts than the frontier and sends nothing anywhere."},"body":{"format":"markdown","text":"## Why this is possible at all\n\nThe first module told you a model is a file of numbers. The history module told you the frontier files are hundreds of gigabytes and need racks of data-centre chips. Between those two facts sits the thing this lesson is about: small open-weights models, made smaller still, that run on the machine you already own.\n\nTwo things make it work. The lesson on scaling explained the first: since 2022 the labs have trained small models on enormous amounts of text, so that an eight-billion-parameter model of 2024 does what a hundred-and-seventy-five-billion-parameter model did in 2020. The second is called quantisation, and it is simple to picture.\n\n## Shrinking the numbers\n\nEach parameter is stored, in training, as a sixteen-bit number: precise to about four decimal places. It turns out the model barely needs that precision to run. Round each number to one of sixteen levels (four bits) and the output is almost indistinguishable, while the file is a quarter of the size. An eight-billion-parameter model is sixteen gigabytes at full precision and under five at four bits, which fits in the memory of a laptop with eight gigabytes and leaves room for the operating system.\n\nQuality does drop, a little, and drops more sharply below four bits. It is the cheapest trade in the field: a few per cent of benchmark score for a machine that costs nothing per query and holds nothing but your own text.\n\n## The tools\n\nThree free tools do nearly all of it, and they are built on the same open-source engine.\n\n**Ollama** is a command-line tool for Windows, Mac and Linux. Install it, and one line downloads and runs a model:\n\n```bash\nollama run gemma3:4b\n```\n\nType, and it answers. Swap the name for any of a few hundred models in its library; `llama3.2`, `qwen3`, `phi4` and `mistral` are the commonest starting points. A second line turns it into a local server that any application on your machine can talk to as if it were a cloud chatbot.\n\n**LM Studio** does the same with a graphical interface: a search box, a download button, a chat window, and a readout of how much memory each model needs before you fetch it. It is the friendlier first door.\n\n**On a phone,** open-source apps run models of one to four billion parameters directly on the handset. A one-billion-parameter model is about half a gigabyte at four bits and answers at reading speed on a mid-range phone from the last few years; a four-billion one needs a recent flagship. Google publishes an open-source gallery app for exactly this on Android.\n\n## What speed to expect\n\nA model generates one token at a time, and on a laptop without a graphics card the limit is how fast the processor can stream the whole file from memory for every token. An eight-billion-parameter model at four bits produces around five to ten tokens a second on a recent laptop processor, which is slower than you read but faster than you type. Apple's recent laptops, whose memory is shared with a capable graphics unit, run the same model at twenty to fifty tokens a second. A gaming card with eight gigabytes of memory runs it faster still. Anything larger than about fourteen billion parameters is a poor experience without a dedicated card.\n\n## What a small model is good for\n\nEverything in the reliable column of the first lesson of this module: rewriting, summarising a document you paste, drafting, translating between major languages, explaining code, answering questions about text you supply. It is good enough at those that a great deal of business software now quietly does them on-device.\n\nWhat it is worse at is everything that was already unreliable, made more so. A four-billion-parameter model has a fraction of the frontier's memory of facts, so invention in specifics is more frequent and starts earlier. Its reasoning over long chains is weaker. Its context window is usually shorter. And in most Indian languages it is markedly weaker than in English, for the reasons the fourth module gave, though several Indian labs publish small open models tuned for Hindi and other languages that improve on it.\n\n## Why bother\n\nThree reasons, each of which the earlier lessons set up. Nothing leaves the machine, so the entire previous-but-one lesson on where your words go does not apply. The model cannot change under you: the file you downloaded is the file you run, next year too. And it works with no connection and no bill, which for a great many people on a great many networks is not a small thing.\n\nThe frontier will always be somewhere else. But the year-old frontier, in your own hands, for nothing, is a new fact about the world, and it is one that the companies selling the frontier do not advertise."},"check":{"question":"An eight-billion-parameter model is sixteen gigabytes as trained and runs on a laptop with eight gigabytes of memory. What made that possible?","options":[{"text":"The laptop streams the model from disk a piece at a time as each token is generated, so it never needs to fit in memory at once.","correct":false,"tempting":"Would make every token take seconds; the model is in memory, at a quarter of its trained size."},{"text":"Each number was rounded from sixteen bits to four, cutting the file to under five gigabytes at a small cost in quality.","correct":true},{"text":"The model was retrained with fewer parameters for laptop use.","correct":false,"tempting":"Confuses quantisation with a smaller model; the parameter count is unchanged and each parameter is stored more coarsely."},{"text":"Half of the model's parameters were deleted, keeping only the important ones.","correct":false,"tempting":"Describes pruning, which is a different and much rarer technique with larger quality costs."}]},"prev":{"slug":"getting-it-for-free","title":"Free tiers, and what they quietly ration","url":"https://addaly.com/learn/ai-actually-explained/getting-it-for-free"},"next":{"slug":"what-a-query-costs","title":"What a query costs, in rupees and watt-hours","url":"https://addaly.com/learn/ai-actually-explained/what-a-query-costs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A model on your own laptop or phone', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/running-one-on-your-own-machine (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A model on your own laptop or phone","courseTitle":"AI, Actually Explained","module":{"index":7,"slug":"putting-it-to-work","title":"Putting it to work without getting burned"},"answer":null,"keyIdea":"Shrinking each of a model’s numbers from sixteen bits to four makes an eight-billion-parameter model fit in five gigabytes and run on an ordinary laptop, where it is slower and thinner on facts than the frontier and sends nothing anywhere.","hero":{"kind":"plate","image":"https://addaly.com/art/modules/ai-actually-explained/putting-it-to-work.webp","alt":"A coaching-centre desk with a stack of books, a phone, an open folder of notes, a steel tumbler and a plain wall calendar above.","caption":"Using these tools well is mostly a matter of what you put on the desk beside them and what you decide never to hand over."},"blocks":[{"t":"h2","id":"why-this-is-possible-at-all","runs":[{"text":"Why this is possible at all"}]},{"t":"p","runs":[{"text":"The first module told you a model is a file of numbers. The history module told you the frontier files are hundreds of gigabytes and need racks of data-centre chips. Between those two facts sits the thing this lesson is about: small open-weights models, made smaller still, that run on the machine you already own."}]},{"t":"p","runs":[{"text":"Two things make it work. The lesson on scaling explained the first: since 2022 the labs have trained small models on enormous amounts of text, so that an eight-billion-parameter model of 2024 does what a hundred-and-seventy-five-billion-parameter model did in 2020. The second is called quantisation, and it is simple to picture."}]},{"t":"h2","id":"shrinking-the-numbers","runs":[{"text":"Shrinking the numbers"}]},{"t":"p","runs":[{"text":"Each parameter is stored, in training, as a sixteen-bit number: precise to about four decimal places. It turns out the model barely needs that precision to run. Round each number to one of sixteen levels (four bits) and the output is almost indistinguishable, while the file is a quarter of the size. An eight-billion-parameter model is sixteen gigabytes at full precision and under five at four bits, which fits in the memory of a laptop with eight gigabytes and leaves room for the operating system."}]},{"t":"p","runs":[{"text":"Quality does drop, a little, and drops more sharply below four bits. It is the cheapest trade in the field: a few per cent of benchmark score for a machine that costs nothing per query and holds nothing but your own text."}]},{"t":"h2","id":"the-tools","runs":[{"text":"The tools"}]},{"t":"p","runs":[{"text":"Three free tools do nearly all of it, and they are built on the same open-source engine."}]},{"t":"p","runs":[{"text":"Ollama","b":true},{"text":" is a command-line tool for Windows, Mac and Linux. Install it, and one line downloads and runs a model:"}]},{"t":"code","lang":"bash","label":"Bash","text":"ollama run gemma3:4b"},{"t":"p","runs":[{"text":"Type, and it answers. Swap the name for any of a few hundred models in its library; "},{"text":"llama3.2","code":true},{"text":", "},{"text":"qwen3","code":true},{"text":", "},{"text":"phi4","code":true},{"text":" and "},{"text":"mistral","code":true},{"text":" are the commonest starting points. A second line turns it into a local server that any application on your machine can talk to as if it were a cloud chatbot."}]},{"t":"p","runs":[{"text":"LM Studio","b":true},{"text":" does the same with a graphical interface: a search box, a download button, a chat window, and a readout of how much memory each model needs before you fetch it. It is the friendlier first door."}]},{"t":"p","runs":[{"text":"On a phone,","b":true},{"text":" open-source apps run models of one to four billion parameters directly on the handset. A one-billion-parameter model is about half a gigabyte at four bits and answers at reading speed on a mid-range phone from the last few years; a four-billion one needs a recent flagship. Google publishes an open-source gallery app for exactly this on Android."}]},{"t":"h2","id":"what-speed-to-expect","runs":[{"text":"What speed to expect"}]},{"t":"p","runs":[{"text":"A model generates one token at a time, and on a laptop without a graphics card the limit is how fast the processor can stream the whole file from memory for every token. An eight-billion-parameter model at four bits produces around five to ten tokens a second on a recent laptop processor, which is slower than you read but faster than you type. Apple’s recent laptops, whose memory is shared with a capable graphics unit, run the same model at twenty to fifty tokens a second. A gaming card with eight gigabytes of memory runs it faster still. Anything larger than about fourteen billion parameters is a poor experience without a dedicated card."}]},{"t":"h2","id":"what-a-small-model-is-good-for","runs":[{"text":"What a small model is good for"}]},{"t":"p","runs":[{"text":"Everything in the reliable column of the first lesson of this module: rewriting, summarising a document you paste, drafting, translating between major languages, explaining code, answering questions about text you supply. It is good enough at those that a great deal of business software now quietly does them on-device."}]},{"t":"p","runs":[{"text":"What it is worse at is everything that was already unreliable, made more so. A four-billion-parameter model has a fraction of the frontier’s memory of facts, so invention in specifics is more frequent and starts earlier. Its reasoning over long chains is weaker. Its context window is usually shorter. And in most Indian languages it is markedly weaker than in English, for the reasons the fourth module gave, though several Indian labs publish small open models tuned for Hindi and other languages that improve on it."}]},{"t":"h2","id":"why-bother","runs":[{"text":"Why bother"}]},{"t":"p","runs":[{"text":"Three reasons, each of which the earlier lessons set up. Nothing leaves the machine, so the entire previous-but-one lesson on where your words go does not apply. The model cannot change under you: the file you downloaded is the file you run, next year too. And it works with no connection and no bill, which for a great many people on a great many networks is not a small thing."}]},{"t":"p","runs":[{"text":"The frontier will always be somewhere else. But the year-old frontier, in your own hands, for nothing, is a new fact about the world, and it is one that the companies selling the frontier do not advertise."}]}],"check":{"question":[{"text":"An eight-billion-parameter model is sixteen gigabytes as trained and runs on a laptop with eight gigabytes of memory. What made that possible?"}],"options":[{"text":[{"text":"The laptop streams the model from disk a piece at a time as each token is generated, so it never needs to fit in memory at once."}],"correct":false,"tempting":[{"text":"Would make every token take seconds; the model is in memory, at a quarter of its trained size."}]},{"text":[{"text":"Each number was rounded from sixteen bits to four, cutting the file to under five gigabytes at a small cost in quality."}],"correct":true,"tempting":null},{"text":[{"text":"The model was retrained with fewer parameters for laptop use."}],"correct":false,"tempting":[{"text":"Confuses quantisation with a smaller model; the parameter count is unchanged and each parameter is stored more coarsely."}]},{"text":[{"text":"Half of the model’s parameters were deleted, keeping only the important ones."}],"correct":false,"tempting":[{"text":"Describes pruning, which is a different and much rarer technique with larger quality costs."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"getting-it-for-free","title":"Free tiers, and what they quietly ration","minutes":8},"next":{"slug":"what-a-query-costs","title":"What a query costs, in rupees and watt-hours","minutes":9}}}