{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"picking-a-size","title":"What each size class can actually do","url":"https://addaly.com/learn/running-models-yourself/picking-a-size","position":4,"of":82,"minutes":10,"answer":"Choose a size class by the job's sensitivity to size (classification and extraction barely care.","keyIdea":"Choose a size class by the job's sensitivity to size (classification and extraction barely care, multi-step reasoning and tool loops care enormously) and try a newer or task-specialised model in the same class before reaching for a bigger one."},"body":{"format":"markdown","text":"## Size classes, described by capability\n\nParameter count is the crudest possible predictor of quality and still the most useful one. Within a generation, and holding training quality roughly constant, here is what each class does. These descriptions age (a good 4B model in 2026 does what an 8B did two years earlier) but the ordering does not.\n\n**Under 1B.** Classification, sentiment, intent routing, keyword extraction, spelling and grammar repair, simple named-entity work. Runs on a phone. Do not ask it to write anything anyone will read.\n\n**1B to 3B.** Summarising a paragraph, rewriting for tone, structured extraction from short text, simple question answering with the answer already in the prompt. Reliable at short, well-specified jobs. Loses the thread after about three instructions.\n\n**4B to 8B.** The workhorse band, and where most local deployments live. Multi-paragraph summaries, decent translation for major languages, single-function code, tagging, drafting, RAG answers over retrieved passages, and reasonable instruction following at up to about eight or ten constraints. Fits on almost any GPU at 4-bit.\n\n**12B to 14B.** A visible step up in instruction following and in code that spans more than one function. Fits in 12 GB at 4-bit with modest context. The best quality-per-gigabyte point on consumer hardware for many people.\n\n**27B to 32B.** Approaching useful general assistant behaviour. Handles ambiguity, holds a longer specification, writes code that compiles more often than not. Needs 24 GB at 4-bit and does not leave much room for context.\n\n**70B and up.** Genuinely capable across most non-frontier work. Needs two 24 GB cards, a 64 GB Mac, or heavy CPU offload and patience.\n\n**Mixture-of-experts models** break this ranking, which is the next lesson.\n\n## Match the class to the job, not to the budget\n\nMost disappointment comes from asking a size class for a capability it does not have, then blaming quantisation or hardware.\n\n> Figure: Two lists that decide whether local is a compromise. Barely notices model size: Classification into a small set of labels; Extracting fields present in the text; Rewriting for tone or for length; Translating between high-resource languages; Generating embeddings for retrieval. Size decides the answer: Reasoning across more than two or three steps; Code that spans several files; Following a fifteen-clause specification; Tool calling in a loop without supervision; Anything where a plausible wrong answer costs. On the left a 3B does almost as well as a 70B, the information needed is in the prompt, the output is short and there is one step: this is where a local model is the correct engineering choice rather than a concession. If your work is entirely on the right, the honest question is whether local is the right approach at all.\n\nJobs that are **size-insensitive** (a 3B does them almost as well as a 70B) are worth naming, because these are where local wins outright: classification into a small label set, extracting fields when the fields are present in the text, boilerplate rewriting, translating simple text between high-resource languages, and generating embeddings.\n\nJobs that are **strongly size-sensitive**: multi-step reasoning, code across several files, following a fifteen-clause specification exactly, reliable tool calling in a loop, and anything where a plausible wrong answer is worse than no answer.\n\nIf your work is entirely in the first list, buy nothing. If it is in the second, the honest question is whether local is the right approach at all.\n\n## The cheapest real improvement\n\nBefore going up a size class, three things usually gain more:\n\n1. **A newer model in the same class.** The gap between model generations at fixed size has been larger than the gap between adjacent sizes in the same generation. A current 8B routinely beats a two-year-old 13B.\n2. **A model trained for your task.** A code-specialised 7B beats a general 32B at code completion, and is four times cheaper to run. The same is true for models trained heavily on a particular language.\n3. **Fixing the prompt and the template.** Free, and it is the actual problem more often than people expect.\n\n## Multilingual reality\n\nModel families differ enormously here and the card rarely says so plainly. Qwen models are strong across Chinese and reasonable across many Asian languages; Gemma has had broad multilingual training; Llama's non-English quality varies by language and drops sharply outside its listed set; several European projects target specific language families.\n\nFor Indian languages specifically, general models handle Hindi in Devanagari passably at 8B and struggle badly with transliterated Hinglish, with Bengali, Tamil, Telugu and Marathi, and with code-switching mid-sentence. Purpose-built projects exist and are worth searching for by language name rather than assuming the famous model is best. Quantisation hurts these languages first, because they occupy the rarest parts of the weight distribution, so a 4-bit quant that is invisible in English can be clearly worse in Marathi.\n\nTest in the language you will actually use. An English benchmark tells you nothing about this.\n\n## A rule for choosing under memory pressure\n\nGiven a fixed amount of memory, prefer more parameters at lower precision down to roughly 4 bits, then stop. A 14B at Q4 in 9 GB beats an 8B at Q8 in 9 GB on most work. Below 4 bits the advantage erodes, and by 2 bits a smaller model at a higher bit rate is usually better, unless the model is very large, where even a crushed 70B holds up. That single ordering resolves most \"which model for my card\" questions without any further argument."},"check":{"question":"A team runs invoice field extraction locally on a 4 GB card with a 3B model and gets 96% field accuracy. They are offered budget for a 24 GB card to run a 32B model. What is the strongest argument against spending it?","options":[{"text":"The 32B would be slower per token, and extraction volume is what matters most in document work.","correct":false,"tempting":"Picks a real but secondary cost; throughput can be bought, whereas a missing quality gap cannot be spent into existence."},{"text":"Extraction is size-insensitive when the fields are present in the text, so most of the remaining 4% is document quality and layout, which more parameters do not fix.","correct":true},{"text":"A 32B at 4-bit would not fit in 24 GB alongside any usable context window, so the upgrade cannot deliver the model they want.","correct":false,"tempting":"States a memory claim that is roughly false (a 32B at Q4 is about 19 GB and fits with short context) and would be an argument for a bigger card rather than none."},{"text":"Larger models hallucinate more on structured output because they are less constrained by the input text.","correct":false,"tempting":"Inverts a real finding; larger models generally follow structure better, and the myth mistakes verbosity for unfaithfulness."}]},"prev":{"slug":"reading-a-model-card","title":"Reading a model card without being sold to","url":"https://addaly.com/learn/running-models-yourself/reading-a-model-card"},"next":{"slug":"mixture-of-experts-locally","title":"Mixture of experts, and why it changes the sum","url":"https://addaly.com/learn/running-models-yourself/mixture-of-experts-locally"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What each size class can actually do', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/picking-a-size (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What each size class can actually do","courseTitle":"Running Models Yourself","module":{"index":1,"slug":"deciding-to-run-it-yourself","title":"Deciding to run it yourself"},"answer":null,"keyIdea":"Choose a size class by the job’s sensitivity to size (classification and extraction barely care, multi-step reasoning and tool loops care enormously) and try a newer or task-specialised model in the same class before reaching for a bigger one.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Barely notices model size","items":["Classification into a small set of labels","Extracting fields present in the text","Rewriting for tone or for length","Translating between high-resource languages","Generating embeddings for retrieval"]},"type":"compare","right":{"title":"Size decides the answer","items":["Reasoning across more than two or three steps","Code that spans several files","Following a fifteen-clause specification","Tool calling in a loop without supervision","Anything where a plausible wrong answer costs"]},"title":"Two lists that decide whether local is a compromise","caption":"On the left a 3B does almost as well as a 70B, the information needed is in the prompt, the output is short and there is one step: this is where a local model is the correct engineering choice rather than a concession. If your work is entirely on the right, the honest question is whether local is the right approach at all."},"alt":"Figure: Two lists that decide whether local is a compromise. Barely notices model size: Classification into a small set of labels; Extracting fields present in the text; Rewriting for tone or for length; Translating between high-resource languages; Generating embeddings for retrieval. Size decides the answer: Reasoning across more than two or three steps; Code that spans several files; Following a fifteen-clause specification; Tool calling in a loop without supervision; Anything where a plausible wrong answer costs. On the left a 3B does almost as well as a 70B, the information needed is in the prompt, the output is short and there is one step: this is where a local model is the correct engineering choice rather than a concession. If your work is entirely on the right, the honest question is whether local is the right approach at all."}},"blocks":[{"t":"h2","id":"size-classes-described-by-capability","runs":[{"text":"Size classes, described by capability"}]},{"t":"p","runs":[{"text":"Parameter count is the crudest possible predictor of quality and still the most useful one. Within a generation, and holding training quality roughly constant, here is what each class does. These descriptions age (a good 4B model in 2026 does what an 8B did two years earlier) but the ordering does not."}]},{"t":"p","runs":[{"text":"Under 1B.","b":true},{"text":" Classification, sentiment, intent routing, keyword extraction, spelling and grammar repair, simple named-entity work. Runs on a phone. Do not ask it to write anything anyone will read."}]},{"t":"p","runs":[{"text":"1B to 3B.","b":true},{"text":" Summarising a paragraph, rewriting for tone, structured extraction from short text, simple question answering with the answer already in the prompt. Reliable at short, well-specified jobs. Loses the thread after about three instructions."}]},{"t":"p","runs":[{"text":"4B to 8B.","b":true},{"text":" The workhorse band, and where most local deployments live. Multi-paragraph summaries, decent translation for major languages, single-function code, tagging, drafting, RAG answers over retrieved passages, and reasonable instruction following at up to about eight or ten constraints. Fits on almost any GPU at 4-bit."}]},{"t":"p","runs":[{"text":"12B to 14B.","b":true},{"text":" A visible step up in instruction following and in code that spans more than one function. Fits in 12 GB at 4-bit with modest context. The best quality-per-gigabyte point on consumer hardware for many people."}]},{"t":"p","runs":[{"text":"27B to 32B.","b":true},{"text":" Approaching useful general assistant behaviour. Handles ambiguity, holds a longer specification, writes code that compiles more often than not. Needs 24 GB at 4-bit and does not leave much room for context."}]},{"t":"p","runs":[{"text":"70B and up.","b":true},{"text":" Genuinely capable across most non-frontier work. Needs two 24 GB cards, a 64 GB Mac, or heavy CPU offload and patience."}]},{"t":"p","runs":[{"text":"Mixture-of-experts models","b":true},{"text":" break this ranking, which is the next lesson."}]},{"t":"h2","id":"match-the-class-to-the-job-not-to-the-budget","runs":[{"text":"Match the class to the job, not to the budget"}]},{"t":"p","runs":[{"text":"Most disappointment comes from asking a size class for a capability it does not have, then blaming quantisation or hardware."}]},{"t":"p","runs":[{"text":"Jobs that are "},{"text":"size-insensitive","b":true},{"text":" (a 3B does them almost as well as a 70B) are worth naming, because these are where local wins outright: classification into a small label set, extracting fields when the fields are present in the text, boilerplate rewriting, translating simple text between high-resource languages, and generating embeddings."}]},{"t":"p","runs":[{"text":"Jobs that are "},{"text":"strongly size-sensitive","b":true},{"text":": multi-step reasoning, code across several files, following a fifteen-clause specification exactly, reliable tool calling in a loop, and anything where a plausible wrong answer is worse than no answer."}]},{"t":"p","runs":[{"text":"If your work is entirely in the first list, buy nothing. If it is in the second, the honest question is whether local is the right approach at all."}]},{"t":"h2","id":"the-cheapest-real-improvement","runs":[{"text":"The cheapest real improvement"}]},{"t":"p","runs":[{"text":"Before going up a size class, three things usually gain more:"}]},{"t":"ol","start":1,"items":[[{"text":"A newer model in the same class.","b":true},{"text":" The gap between model generations at fixed size has been larger than the gap between adjacent sizes in the same generation. A current 8B routinely beats a two-year-old 13B."}],[{"text":"A model trained for your task.","b":true},{"text":" A code-specialised 7B beats a general 32B at code completion, and is four times cheaper to run. The same is true for models trained heavily on a particular language."}],[{"text":"Fixing the prompt and the template.","b":true},{"text":" Free, and it is the actual problem more often than people expect."}]]},{"t":"h2","id":"multilingual-reality","runs":[{"text":"Multilingual reality"}]},{"t":"p","runs":[{"text":"Model families differ enormously here and the card rarely says so plainly. Qwen models are strong across Chinese and reasonable across many Asian languages; Gemma has had broad multilingual training; Llama’s non-English quality varies by language and drops sharply outside its listed set; several European projects target specific language families."}]},{"t":"p","runs":[{"text":"For Indian languages specifically, general models handle Hindi in Devanagari passably at 8B and struggle badly with transliterated Hinglish, with Bengali, Tamil, Telugu and Marathi, and with code-switching mid-sentence. Purpose-built projects exist and are worth searching for by language name rather than assuming the famous model is best. Quantisation hurts these languages first, because they occupy the rarest parts of the weight distribution, so a 4-bit quant that is invisible in English can be clearly worse in Marathi."}]},{"t":"p","runs":[{"text":"Test in the language you will actually use. An English benchmark tells you nothing about this."}]},{"t":"h2","id":"a-rule-for-choosing-under-memory-pressure","runs":[{"text":"A rule for choosing under memory pressure"}]},{"t":"p","runs":[{"text":"Given a fixed amount of memory, prefer more parameters at lower precision down to roughly 4 bits, then stop. A 14B at Q4 in 9 GB beats an 8B at Q8 in 9 GB on most work. Below 4 bits the advantage erodes, and by 2 bits a smaller model at a higher bit rate is usually better, unless the model is very large, where even a crushed 70B holds up. That single ordering resolves most “which model for my card” questions without any further argument."}]}],"check":{"question":[{"text":"A team runs invoice field extraction locally on a 4 GB card with a 3B model and gets 96% field accuracy. They are offered budget for a 24 GB card to run a 32B model. What is the strongest argument against spending it?"}],"options":[{"text":[{"text":"The 32B would be slower per token, and extraction volume is what matters most in document work."}],"correct":false,"tempting":[{"text":"Picks a real but secondary cost; throughput can be bought, whereas a missing quality gap cannot be spent into existence."}]},{"text":[{"text":"Extraction is size-insensitive when the fields are present in the text, so most of the remaining 4% is document quality and layout, which more parameters do not fix."}],"correct":true,"tempting":null},{"text":[{"text":"A 32B at 4-bit would not fit in 24 GB alongside any usable context window, so the upgrade cannot deliver the model they want."}],"correct":false,"tempting":[{"text":"States a memory claim that is roughly false (a 32B at Q4 is about 19 GB and fits with short context) and would be an argument for a bigger card rather than none."}]},{"text":[{"text":"Larger models hallucinate more on structured output because they are less constrained by the input text."}],"correct":false,"tempting":[{"text":"Inverts a real finding; larger models generally follow structure better, and the myth mistakes verbosity for unfaithfulness."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"reading-a-model-card","title":"Reading a model card without being sold to","minutes":9},"next":{"slug":"mixture-of-experts-locally","title":"Mixture of experts, and why it changes the sum","minutes":9}}}