{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"why-run-a-model-yourself","title":"Why run a model yourself, and when not to","url":"https://addaly.com/learn/running-models-yourself/why-run-a-model-yourself","position":1,"of":82,"minutes":6,"answer":"The weights set the quality ceiling; hardware only decides whether you can reach it and how fast.","keyIdea":"The weights set the quality ceiling; hardware only decides whether you can reach it and how fast."},"body":{"format":"markdown","text":"Almost everything written about local models is either an advertisement or an install guide. Start with the question those skip: should you do this at all?\n\n## What running it yourself actually buys\n\n**The data never leaves.** Clinical notes, legal discovery, an employer's source code, other people's private messages. No processing agreement, no sub-processor list, no question about training on your inputs. For some work this is not a preference, it is the condition of doing the work at all.\n\n**No rate limits, no queue, no account.** You can hammer it for six hours. Nobody suspends you at 2am mid-batch.\n\n**It works with no network.** On a plane, on a ship, in a lab with no outbound access, in a country where the API is blocked or the payment method is not accepted. This last one is not a small case.\n\n**Nobody deprecates it.** Hosted models get retired with a few months' notice, and the replacement behaves differently. A GGUF file on your disk behaves identically in five years. If you have tuned prompts around a specific model's quirks, that stability is worth real money.\n\n**Cost, but only under conditions.** Local can be dramatically cheaper or dramatically more expensive than an API depending on one variable. Lesson 10 does that arithmetic properly. Do not assume it yet.\n\n## The honest reasons not to\n\n**The quality gap is real and it is large.** An 8B model at 4-bit is a capable 2023-class assistant. It summarises, classifies, extracts, rewrites, and writes short functions well. It does not do long multi-step reasoning, does not hold a large codebase in its head, and calls tools less reliably. No amount of hardware changes this: the weights set the ceiling.\n\n**Your time is the largest line item.** An afternoon spent debugging a chat template costs more than a year of light API use at any professional rate.\n\n**Speed, especially before the first token.** On a CPU, generation might be tolerable while processing a 2,000-token prompt takes 30 to 60 seconds. That delay, not tokens per second, is what makes people give up.\n\n**You are the operations team.** When it stops working at 3am, that is your problem.\n\n## A test that decides it in an hour\n\nDo not start by buying hardware. Separate two questions that people fuse together: *is this model good enough for my task?* and *can I run it?*\n\nTake twenty real prompts from your actual work. Run them against a hosted copy of the exact open weights you would run locally: most open models are available through several providers for a few cents. Grade the outputs yourself.\n\nIf the quality is unacceptable there, it will be unacceptable locally, and a bigger GPU will not fix it. If it is acceptable, you now know the target and can shop for the cheapest way to run those specific weights.\n\nThis costs about a dollar and it is the single most useful thing in this course.\n\n## What you will need to be able to do\n\nThe rest of these lessons assume you want to answer four questions without asking anyone:\n\n1. Given any model on Hugging Face, does it fit in my memory, and at what context length?\n2. What did quantisation cost me, and where will I notice?\n3. Which serving engine wins for my pattern of use, and by what measure of \"wins\"?\n4. Is this cheaper than the API for my volume?\n\nThose are arithmetic and engineering questions with real answers. Almost nothing else in this field is."},"check":{"question":"Someone runs Llama 3.1 8B locally and finds it noticeably worse than the same model served by a hosted provider. They conclude they need a bigger GPU. What is the best response?","options":[{"text":"Same weights have the same ceiling, so check quantisation level, chat template, context truncation and sampler settings before buying anything.","correct":true},{"text":"A larger GPU gives the model more capacity to work with, so quality does improve with more VRAM.","correct":false,"tempting":"Treats VRAM as if it were model capability, when it only determines what you can load and how fast."},{"text":"Providers serve a larger private model behind the public name, so a local copy can never match it.","correct":false,"tempting":"Reaches for an unfalsifiable explanation instead of the four testable causes sitting in front of you."},{"text":"Quantisation always costs a large share of quality, so a local model is inherently the weaker one.","correct":false,"tempting":"Assumes 4-bit is a big loss, when a well-made 4-bit quant is usually a small and measurable one."}]},"prev":null,"next":{"slug":"the-open-weights-landscape","title":"What \"open\" means, and what it does not","url":"https://addaly.com/learn/running-models-yourself/the-open-weights-landscape"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why run a model yourself, and when not to', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/why-run-a-model-yourself (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why run a model yourself, and when not to","courseTitle":"Running Models Yourself","module":{"index":1,"slug":"deciding-to-run-it-yourself","title":"Deciding to run it yourself"},"answer":null,"keyIdea":"The weights set the quality ceiling; hardware only decides whether you can reach it and how fast.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"p","runs":[{"text":"Almost everything written about local models is either an advertisement or an install guide. Start with the question those skip: should you do this at all?"}],"lede":true},{"t":"h2","id":"what-running-it-yourself-actually-buys","runs":[{"text":"What running it yourself actually buys"}]},{"t":"p","runs":[{"text":"The data never leaves.","b":true},{"text":" Clinical notes, legal discovery, an employer’s source code, other people’s private messages. No processing agreement, no sub-processor list, no question about training on your inputs. For some work this is not a preference, it is the condition of doing the work at all."}]},{"t":"p","runs":[{"text":"No rate limits, no queue, no account.","b":true},{"text":" You can hammer it for six hours. Nobody suspends you at 2am mid-batch."}]},{"t":"p","runs":[{"text":"It works with no network.","b":true},{"text":" On a plane, on a ship, in a lab with no outbound access, in a country where the API is blocked or the payment method is not accepted. This last one is not a small case."}]},{"t":"p","runs":[{"text":"Nobody deprecates it.","b":true},{"text":" Hosted models get retired with a few months’ notice, and the replacement behaves differently. A GGUF file on your disk behaves identically in five years. If you have tuned prompts around a specific model’s quirks, that stability is worth real money."}]},{"t":"p","runs":[{"text":"Cost, but only under conditions.","b":true},{"text":" Local can be dramatically cheaper or dramatically more expensive than an API depending on one variable. Lesson 10 does that arithmetic properly. Do not assume it yet."}]},{"t":"h2","id":"the-honest-reasons-not-to","runs":[{"text":"The honest reasons not to"}]},{"t":"p","runs":[{"text":"The quality gap is real and it is large.","b":true},{"text":" An 8B model at 4-bit is a capable 2023-class assistant. It summarises, classifies, extracts, rewrites, and writes short functions well. It does not do long multi-step reasoning, does not hold a large codebase in its head, and calls tools less reliably. No amount of hardware changes this: the weights set the ceiling."}]},{"t":"p","runs":[{"text":"Your time is the largest line item.","b":true},{"text":" An afternoon spent debugging a chat template costs more than a year of light API use at any professional rate."}]},{"t":"p","runs":[{"text":"Speed, especially before the first token.","b":true},{"text":" On a CPU, generation might be tolerable while processing a 2,000-token prompt takes 30 to 60 seconds. That delay, not tokens per second, is what makes people give up."}]},{"t":"p","runs":[{"text":"You are the operations team.","b":true},{"text":" When it stops working at 3am, that is your problem."}]},{"t":"h2","id":"a-test-that-decides-it-in-an-hour","runs":[{"text":"A test that decides it in an hour"}]},{"t":"p","runs":[{"text":"Do not start by buying hardware. Separate two questions that people fuse together: "},{"text":"is this model good enough for my task?","i":true},{"text":" and "},{"text":"can I run it?","i":true}]},{"t":"p","runs":[{"text":"Take twenty real prompts from your actual work. Run them against a hosted copy of the exact open weights you would run locally: most open models are available through several providers for a few cents. Grade the outputs yourself."}]},{"t":"p","runs":[{"text":"If the quality is unacceptable there, it will be unacceptable locally, and a bigger GPU will not fix it. If it is acceptable, you now know the target and can shop for the cheapest way to run those specific weights."}]},{"t":"p","runs":[{"text":"This costs about a dollar and it is the single most useful thing in this course."}]},{"t":"h2","id":"what-you-will-need-to-be-able-to-do","runs":[{"text":"What you will need to be able to do"}]},{"t":"p","runs":[{"text":"The rest of these lessons assume you want to answer four questions without asking anyone:"}]},{"t":"ol","start":1,"items":[[{"text":"Given any model on Hugging Face, does it fit in my memory, and at what context length?"}],[{"text":"What did quantisation cost me, and where will I notice?"}],[{"text":"Which serving engine wins for my pattern of use, and by what measure of “wins”?"}],[{"text":"Is this cheaper than the API for my volume?"}]]},{"t":"p","runs":[{"text":"Those are arithmetic and engineering questions with real answers. Almost nothing else in this field is."}]}],"check":{"question":[{"text":"Someone runs Llama 3.1 8B locally and finds it noticeably worse than the same model served by a hosted provider. They conclude they need a bigger GPU. What is the best response?"}],"options":[{"text":[{"text":"Same weights have the same ceiling, so check quantisation level, chat template, context truncation and sampler settings before buying anything."}],"correct":true,"tempting":null},{"text":[{"text":"A larger GPU gives the model more capacity to work with, so quality does improve with more VRAM."}],"correct":false,"tempting":[{"text":"Treats VRAM as if it were model capability, when it only determines what you can load and how fast."}]},{"text":[{"text":"Providers serve a larger private model behind the public name, so a local copy can never match it."}],"correct":false,"tempting":[{"text":"Reaches for an unfalsifiable explanation instead of the four testable causes sitting in front of you."}]},{"text":[{"text":"Quantisation always costs a large share of quality, so a local model is inherently the weaker one."}],"correct":false,"tempting":[{"text":"Assumes 4-bit is a big loss, when a well-made 4-bit quant is usually a small and measurable one."}]}],"order":[2,1,0,3],"why":null},"prev":null,"next":{"slug":"the-open-weights-landscape","title":"What “open” means, and what it does not","minutes":10}}}