{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"choosing-for-the-job","title":"Choosing a Model for the Job","url":"https://addaly.com/learn/open-models/choosing-for-the-job","position":31,"of":71,"minutes":9,"answer":"Fit the constraints first, memory and latency and licence and language, then test the two or three models left standing.","keyIdea":"Fit the constraints first, memory and latency and licence and language, then test the two or three models left standing."},"body":{"format":"markdown","text":"## Write the constraints before you look at candidates\n\nFive numbers, on paper, before any browsing:\n\n1. Memory budget on the machine that will actually run it.\n2. Latency budget at p95, from a real user's point of view.\n3. Requests per day.\n4. Licence constraint (commercial, redistributable, attribution acceptable).\n5. Whether data may leave the building.\n\nMost shortlists collapse to two or three candidates immediately. That is the point.\n\n## The memory arithmetic\n\n**Weights** = parameters × bytes per weight.\n\n- bf16 ≈ 2 bytes → an 8B model is about 16 GB\n- Q8 ≈ 1 byte → about 8.5 GB\n- Q4_K_M ≈ 0.6 bytes → about 4.9 GB\n\n**KV cache** = 2 × layers × kv_heads × head_dim × bytes, per token. For Llama-3.1-8B (32 layers, 8 KV heads, head dim 128, fp16): 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KB per token.\n\n- 8k context → about 1.0 GB\n- 32k context → about 4.2 GB\n\nThat arithmetic is why a model that loads happily at 2k context dies at 32k. Add roughly 1 GB of runtime overhead, and aim for weights + KV + 1 GB under 90% of your budget.\n\n## If you have 8 GB of RAM and no discrete GPU\n\nThis is most people, and most writing about local models pretends otherwise. Honestly:\n\n- **3B to 4B instruct at Q4.** Comfortable. About 2.5 GB of weights, room for 8k context, roughly 8 to 20 tokens per second on a recent CPU. Genuinely good at extraction, classification, tagging, rewriting, short summaries.\n- **7B to 8B at Q4.** Fits at about 4.9 GB, but leaves little for context and nothing for your browser. Expect 4 to 10 tokens per second. Fine for overnight batch work, painful for chat.\n- **14B and above.** Not on that machine. Not \"slow\": not usable.\n\nApple Silicon changes the maths, because unified memory is shared with the GPU and bandwidth is high. A 16 GB M-series laptop behaves roughly like a 14B-at-Q4 machine, which is a real capability difference from a 16 GB x86 laptop with integrated graphics.\n\nOn mixture-of-experts: a 30B model with 3B active still needs the whole 30B resident (around 18 GB at Q4) but computes at roughly 3B speed. Excellent if you have the RAM. Irrelevant if you do not.\n\nAnd renting is not cheating. OpenRouter, Together, Fireworks, DeepInfra and others serve these same open weights per token. The freedom of open weights is that you *could* run it yourself and nobody can take the model away; it does not oblige you to buy a GPU. Self-host when data cannot leave, when volume makes it cheaper, or when you need it offline.\n\n## Match the shape of the task\n\n- **Extraction, classification, routing.** Smallest instruct model that passes. Use JSON schema or grammar-constrained decoding, temperature 0. Model choice matters less than constraints here.\n- **Long documents.** Advertised context is not usable context. Most models degrade well before their maximum. Test retrieval from the middle of your own longest document before believing the number on the card.\n- **Code.** Use the coder-tuned variant and check which languages the card names. The gap between a general and a coder model of the same size is large.\n- **Tool use and agents.** Check the card explicitly claims function-calling training, then test with your real schemas. This capability varies far more between models than chat quality does.\n- **Non-English.** Check the language list, then test. Also check tokenizer efficiency: some models spend two or three times as many tokens on non-Latin scripts, which costs you money and eats your effective context at the same time.\n\n## The shortlist rule\n\nThree candidates. No more than one size class apart. All licence-clear. Then stop reading and start measuring, which is the next lesson."},"check":{"question":"On a 16 GB laptop with no GPU, a 14B model at Q4 (about 8 GB of weights) loads fine but the machine thrashes once 32k context is enabled. Why?","options":[{"text":"The KV cache grows with context length and can reach several gigabytes at 32k, on top of the weights and everything else the OS is holding.","correct":true},{"text":"Quantized loading briefly materialises the fp16 weights in memory, so a Q4 model still needs its full-precision footprint at startup.","correct":false,"tempting":"A plausible-sounding load-time myth; quantized formats are read directly and never expand to full precision in RAM."},{"text":"14B models require a GPU, and CPU inference is not actually supported at that size.","correct":false,"tempting":"Confuses slow with unsupported; CPU inference works at any size, it just gets impractical."},{"text":"Longer context enlarges the embedding and vocabulary tables the model must keep resident.","correct":false,"tempting":"Blames a fixed-size component; embeddings depend on vocabulary size, not on how many tokens you feed in."}]},"prev":{"slug":"stop-conditions","title":"Why it will not stop talking","url":"https://addaly.com/learn/open-models/stop-conditions"},"next":{"slug":"quantisation-explained","title":"What quantisation actually does","url":"https://addaly.com/learn/open-models/quantisation-explained"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Choosing a Model for the Job', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/choosing-for-the-job (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Choosing a Model for the Job","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Fit the constraints first, memory and latency and licence and language, then test the two or three models left standing.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"write-the-constraints-before-you-look-at-candidates","runs":[{"text":"Write the constraints before you look at candidates"}]},{"t":"p","runs":[{"text":"Five numbers, on paper, before any browsing:"}]},{"t":"ol","start":1,"items":[[{"text":"Memory budget on the machine that will actually run it."}],[{"text":"Latency budget at p95, from a real user’s point of view."}],[{"text":"Requests per day."}],[{"text":"Licence constraint (commercial, redistributable, attribution acceptable)."}],[{"text":"Whether data may leave the building."}]]},{"t":"p","runs":[{"text":"Most shortlists collapse to two or three candidates immediately. That is the point."}]},{"t":"h2","id":"the-memory-arithmetic","runs":[{"text":"The memory arithmetic"}]},{"t":"p","runs":[{"text":"Weights","b":true},{"text":" = parameters × bytes per weight."}]},{"t":"ul","items":[[{"text":"bf16 ≈ 2 bytes → an 8B model is about 16 GB"}],[{"text":"Q8 ≈ 1 byte → about 8.5 GB"}],[{"text":"Q4_K_M ≈ 0.6 bytes → about 4.9 GB"}]]},{"t":"p","runs":[{"text":"KV cache","b":true},{"text":" = 2 × layers × kv_heads × head_dim × bytes, per token. For Llama-3.1-8B (32 layers, 8 KV heads, head dim 128, fp16): 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KB per token."}]},{"t":"ul","items":[[{"text":"8k context → about 1.0 GB"}],[{"text":"32k context → about 4.2 GB"}]]},{"t":"p","runs":[{"text":"That arithmetic is why a model that loads happily at 2k context dies at 32k. Add roughly 1 GB of runtime overhead, and aim for weights + KV + 1 GB under 90% of your budget."}]},{"t":"h2","id":"if-you-have-8-gb-of-ram-and-no-discrete-gpu","runs":[{"text":"If you have 8 GB of RAM and no discrete GPU"}]},{"t":"p","runs":[{"text":"This is most people, and most writing about local models pretends otherwise. Honestly:"}]},{"t":"ul","items":[[{"text":"3B to 4B instruct at Q4.","b":true},{"text":" Comfortable. About 2.5 GB of weights, room for 8k context, roughly 8 to 20 tokens per second on a recent CPU. Genuinely good at extraction, classification, tagging, rewriting, short summaries."}],[{"text":"7B to 8B at Q4.","b":true},{"text":" Fits at about 4.9 GB, but leaves little for context and nothing for your browser. Expect 4 to 10 tokens per second. Fine for overnight batch work, painful for chat."}],[{"text":"14B and above.","b":true},{"text":" Not on that machine. Not “slow”: not usable."}]]},{"t":"p","runs":[{"text":"Apple Silicon changes the maths, because unified memory is shared with the GPU and bandwidth is high. A 16 GB M-series laptop behaves roughly like a 14B-at-Q4 machine, which is a real capability difference from a 16 GB x86 laptop with integrated graphics."}]},{"t":"p","runs":[{"text":"On mixture-of-experts: a 30B model with 3B active still needs the whole 30B resident (around 18 GB at Q4) but computes at roughly 3B speed. Excellent if you have the RAM. Irrelevant if you do not."}]},{"t":"p","runs":[{"text":"And renting is not cheating. OpenRouter, Together, Fireworks, DeepInfra and others serve these same open weights per token. The freedom of open weights is that you "},{"text":"could","i":true},{"text":" run it yourself and nobody can take the model away; it does not oblige you to buy a GPU. Self-host when data cannot leave, when volume makes it cheaper, or when you need it offline."}]},{"t":"h2","id":"match-the-shape-of-the-task","runs":[{"text":"Match the shape of the task"}]},{"t":"ul","items":[[{"text":"Extraction, classification, routing.","b":true},{"text":" Smallest instruct model that passes. Use JSON schema or grammar-constrained decoding, temperature 0. Model choice matters less than constraints here."}],[{"text":"Long documents.","b":true},{"text":" Advertised context is not usable context. Most models degrade well before their maximum. Test retrieval from the middle of your own longest document before believing the number on the card."}],[{"text":"Code.","b":true},{"text":" Use the coder-tuned variant and check which languages the card names. The gap between a general and a coder model of the same size is large."}],[{"text":"Tool use and agents.","b":true},{"text":" Check the card explicitly claims function-calling training, then test with your real schemas. This capability varies far more between models than chat quality does."}],[{"text":"Non-English.","b":true},{"text":" Check the language list, then test. Also check tokenizer efficiency: some models spend two or three times as many tokens on non-Latin scripts, which costs you money and eats your effective context at the same time."}]]},{"t":"h2","id":"the-shortlist-rule","runs":[{"text":"The shortlist rule"}]},{"t":"p","runs":[{"text":"Three candidates. No more than one size class apart. All licence-clear. Then stop reading and start measuring, which is the next lesson."}]}],"check":{"question":[{"text":"On a 16 GB laptop with no GPU, a 14B model at Q4 (about 8 GB of weights) loads fine but the machine thrashes once 32k context is enabled. Why?"}],"options":[{"text":[{"text":"The KV cache grows with context length and can reach several gigabytes at 32k, on top of the weights and everything else the OS is holding."}],"correct":true,"tempting":null},{"text":[{"text":"Quantized loading briefly materialises the fp16 weights in memory, so a Q4 model still needs its full-precision footprint at startup."}],"correct":false,"tempting":[{"text":"A plausible-sounding load-time myth; quantized formats are read directly and never expand to full precision in RAM."}]},{"text":[{"text":"14B models require a GPU, and CPU inference is not actually supported at that size."}],"correct":false,"tempting":[{"text":"Confuses slow with unsupported; CPU inference works at any size, it just gets impractical."}]},{"text":[{"text":"Longer context enlarges the embedding and vocabulary tables the model must keep resident."}],"correct":false,"tempting":[{"text":"Blames a fixed-size component; embeddings depend on vocabulary size, not on how many tokens you feed in."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"stop-conditions","title":"Why it will not stop talking","minutes":8},"next":{"slug":"quantisation-explained","title":"What quantisation actually does","minutes":10}}}