{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"small-models-and-edge","title":"Small models, where fine-tuning matters most","url":"https://addaly.com/learn/fine-tuning/small-models-and-edge","position":65,"of":84,"minutes":9,"answer":"Fine-tuning helps more the smaller the model, because a small model has to be given a behaviour rather than prompted into one. And the case for on-device models is privacy.","keyIdea":"Fine-tuning helps more the smaller the model, because a small model has to be given a behaviour rather than prompted into one. And the case for on-device models is privacy, offline operation and zero marginal cost rather than price, against the specific cost that they collapse rather than degrade off-distribution."},"body":{"format":"markdown","text":"## The effect is inverse to size\n\nFine-tuning a frontier-scale model on a narrow task gives a modest improvement, because the model was already close. Fine-tuning a 0.5B model on the same task can transform it, because the base model was barely adequate.\n\nThe mechanism: a small model has a weak prior about how to follow instructions and a narrow range of behaviours it can be prompted into. Prompting asks it to select a behaviour; training gives it one. On a task narrow enough, a 0.5B or 1.5B model tuned on a few thousand examples can match a very large model prompted, for that task, and only that task.\n\nThat qualification is the whole deal. You are trading breadth for a specific competence at a hundredth of the cost.\n\n## What runs where\n\nIn 4-bit quantisation, approximately:\n\n- **0.5B**: about 400 MB. Runs on a mid-range phone.\n- **1.5B**: about 1 GB. Comfortable on a phone, instant on a laptop.\n- **3B**: about 2 GB. Any modern laptop.\n- **7B to 8B**: about 5 GB. A laptop with 8 GB of RAM, at a few tokens a second on CPU and much faster with any GPU or on Apple silicon.\n\nFor training: below about 1.5B, prefer **full fine-tuning** over LoRA. It fits on a free T4 with 8-bit AdamW, it is simpler, and at that size the extra capacity to change is usually what you want rather than something to be constrained.\n\n## The serving stack, all free\n\n- **llama.cpp / GGUF**: CPU and GPU, every desktop platform, and the format most other tools read. Ollama and LM Studio wrap it.\n- **ONNX Runtime**: broad hardware support, good on Windows and on CPU.\n- **ExecuTorch** for iOS and Android, **MLC LLM** for mobile GPUs, **Core ML** on Apple devices.\n\nThe path from a training run to a phone is: train the adapter, merge it, convert to GGUF, quantise, test. All of it free and all of it documented.\n\n## The real reason, which is not cost\n\nSmall on-device models are usually justified on price. That is the weakest argument, because a hosted small model is already cheap.\n\n> Figure: In 4-bit, and where each one runs. Layers, top to bottom: 0.5B: about 400 MB (Runs on a mid-range phone. Fine-tuning transforms it, because the base model was barely adequate.); 1.5B: about 1 GB (Comfortable on a phone, instant on a laptop.); 3B: about 2 GB (Any modern laptop.); 8B: about 5 GB (A laptop with 8 GB of RAM, at a few tokens a second on CPU and much faster with any GPU.). Below about 1.5B, prefer full fine-tuning to LoRA: it fits a free T4 with 8-bit AdamW, and at that size the extra capacity to change is what you want rather than something to constrain.\n\nThe strong reasons are the ones an API cannot offer at any price:\n\n- **The data does not leave the device.** For a health app, a journal, a legal notes tool, this is the product rather than a feature.\n- **It works offline.** On a train, in a rural clinic, on a factory floor.\n- **Latency has no network in it.** 60 milliseconds to first token instead of 400.\n- **There is no per-request cost**, so you can call the model for things that would never justify an API call: ranking every item in a list, checking every keystroke.\n- **No vendor can change or withdraw it.** The model you shipped is the model that runs, for as long as the app exists.\n\n## The limitations, plainly\n\nSmall models fail differently from large ones, and worse:\n\n- **Off-distribution they collapse rather than degrade.** A large model handles an unexpected input awkwardly; a 1B model tuned on triage produces triage-shaped output for a recipe request, confidently.\n- **They have very little world knowledge.** Anything factual must come from the context window. This is not a tuning problem and no dataset fixes it.\n- **Multi-step reasoning is weak.** Chains of inference break down. Decompose the task into single steps, or do not use a small model for it.\n- **Long context is expensive relative to their capacity**: a 2B model with a 32k window will attend over all of it without making much use of what it finds.\n\nSo the design pattern that works is a narrow, well-specified task with the necessary facts supplied in the prompt, plus a route to something larger when the input does not look like the task. The classifier from module one is a good gate for exactly that.\n\n## Where to start\n\nThe small-model families are excellent now and improving: Qwen at 0.5B and 1.5B, Llama at 1B and 3B, Gemma's small sizes, SmolLM at 135M to 1.7B, Phi's small variants. Licences differ (several are Apache-2.0, others carry community terms), so check the one you pick.\n\nAnd test the base model on your task before training. The small models of the last two years are substantially better than their predecessors, and the gap you were going to close with a training run may have closed on its own."},"check":{"question":"A 1B model fine-tuned on ticket triage is sent a recipe question by a user who misunderstood the product. It produces a confident triage classification. Why is this failure mode characteristic of small models rather than large ones?","options":[{"text":"Small models have shorter context windows, so the recipe question was truncated into something resembling a ticket.","correct":false,"tempting":"Reaches for truncation, which would not apply to a short question and does not explain confident misclassification."},{"text":"Quantisation to 4 bits removes the precision needed to represent uncertainty, so all outputs become confident.","correct":false,"tempting":"Attributes the behaviour to quantisation, which affects a full-precision model of the same size identically in this respect."},{"text":"It has a narrow range of behaviours, so training gave it one and it has nothing to fall back on.","correct":true},{"text":"One billion parameters cannot store the refusal behaviour, which requires a minimum model size to be learned at all.","correct":false,"tempting":"Posits a capacity threshold for refusal; small models can be trained to decline, and this one simply was not shown any examples of it."}]},"prev":{"slug":"structure-without-training","title":"Getting structure without training for it","url":"https://addaly.com/learn/fine-tuning/structure-without-training"},"next":{"slug":"tabular-and-time-series","title":"Tabular and time series: the honest answer","url":"https://addaly.com/learn/fine-tuning/tabular-and-time-series"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Small models, where fine-tuning matters most', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/small-models-and-edge (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Small models, where fine-tuning matters most","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"Fine-tuning helps more the smaller the model, because a small model has to be given a behaviour rather than prompted into one. And the case for on-device models is privacy, offline operation and zero marginal cost rather than price, against the specific cost that they collapse rather than degrade off-distribution.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"In 4-bit, and where each one runs","layers":[{"label":"0.5B: about 400 MB","detail":"Runs on a mid-range phone. Fine-tuning transforms it, because the base model was barely adequate."},{"label":"1.5B: about 1 GB","detail":"Comfortable on a phone, instant on a laptop."},{"label":"3B: about 2 GB","detail":"Any modern laptop."},{"label":"8B: about 5 GB","detail":"A laptop with 8 GB of RAM, at a few tokens a second on CPU and much faster with any GPU."}],"caption":"Below about 1.5B, prefer full fine-tuning to LoRA: it fits a free T4 with 8-bit AdamW, and at that size the extra capacity to change is what you want rather than something to constrain."},"alt":"Figure: In 4-bit, and where each one runs. Layers, top to bottom: 0.5B: about 400 MB (Runs on a mid-range phone. Fine-tuning transforms it, because the base model was barely adequate.); 1.5B: about 1 GB (Comfortable on a phone, instant on a laptop.); 3B: about 2 GB (Any modern laptop.); 8B: about 5 GB (A laptop with 8 GB of RAM, at a few tokens a second on CPU and much faster with any GPU.). Below about 1.5B, prefer full fine-tuning to LoRA: it fits a free T4 with 8-bit AdamW, and at that size the extra capacity to change is what you want rather than something to constrain."}},"blocks":[{"t":"h2","id":"the-effect-is-inverse-to-size","runs":[{"text":"The effect is inverse to size"}]},{"t":"p","runs":[{"text":"Fine-tuning a frontier-scale model on a narrow task gives a modest improvement, because the model was already close. Fine-tuning a 0.5B model on the same task can transform it, because the base model was barely adequate."}]},{"t":"p","runs":[{"text":"The mechanism: a small model has a weak prior about how to follow instructions and a narrow range of behaviours it can be prompted into. Prompting asks it to select a behaviour; training gives it one. On a task narrow enough, a 0.5B or 1.5B model tuned on a few thousand examples can match a very large model prompted, for that task, and only that task."}]},{"t":"p","runs":[{"text":"That qualification is the whole deal. You are trading breadth for a specific competence at a hundredth of the cost."}]},{"t":"h2","id":"what-runs-where","runs":[{"text":"What runs where"}]},{"t":"p","runs":[{"text":"In 4-bit quantisation, approximately:"}]},{"t":"ul","items":[[{"text":"0.5B","b":true},{"text":": about 400 MB. Runs on a mid-range phone."}],[{"text":"1.5B","b":true},{"text":": about 1 GB. Comfortable on a phone, instant on a laptop."}],[{"text":"3B","b":true},{"text":": about 2 GB. Any modern laptop."}],[{"text":"7B to 8B","b":true},{"text":": about 5 GB. A laptop with 8 GB of RAM, at a few tokens a second on CPU and much faster with any GPU or on Apple silicon."}]]},{"t":"p","runs":[{"text":"For training: below about 1.5B, prefer "},{"text":"full fine-tuning","b":true},{"text":" over LoRA. It fits on a free T4 with 8-bit AdamW, it is simpler, and at that size the extra capacity to change is usually what you want rather than something to be constrained."}]},{"t":"h2","id":"the-serving-stack-all-free","runs":[{"text":"The serving stack, all free"}]},{"t":"ul","items":[[{"text":"llama.cpp / GGUF","b":true},{"text":": CPU and GPU, every desktop platform, and the format most other tools read. Ollama and LM Studio wrap it."}],[{"text":"ONNX Runtime","b":true},{"text":": broad hardware support, good on Windows and on CPU."}],[{"text":"ExecuTorch","b":true},{"text":" for iOS and Android, "},{"text":"MLC LLM","b":true},{"text":" for mobile GPUs, "},{"text":"Core ML","b":true},{"text":" on Apple devices."}]]},{"t":"p","runs":[{"text":"The path from a training run to a phone is: train the adapter, merge it, convert to GGUF, quantise, test. All of it free and all of it documented."}]},{"t":"h2","id":"the-real-reason-which-is-not-cost","runs":[{"text":"The real reason, which is not cost"}]},{"t":"p","runs":[{"text":"Small on-device models are usually justified on price. That is the weakest argument, because a hosted small model is already cheap."}]},{"t":"p","runs":[{"text":"The strong reasons are the ones an API cannot offer at any price:"}]},{"t":"ul","items":[[{"text":"The data does not leave the device.","b":true},{"text":" For a health app, a journal, a legal notes tool, this is the product rather than a feature."}],[{"text":"It works offline.","b":true},{"text":" On a train, in a rural clinic, on a factory floor."}],[{"text":"Latency has no network in it.","b":true},{"text":" 60 milliseconds to first token instead of 400."}],[{"text":"There is no per-request cost","b":true},{"text":", so you can call the model for things that would never justify an API call: ranking every item in a list, checking every keystroke."}],[{"text":"No vendor can change or withdraw it.","b":true},{"text":" The model you shipped is the model that runs, for as long as the app exists."}]]},{"t":"h2","id":"the-limitations-plainly","runs":[{"text":"The limitations, plainly"}],"role":"limit"},{"t":"p","runs":[{"text":"Small models fail differently from large ones, and worse:"}]},{"t":"ul","items":[[{"text":"Off-distribution they collapse rather than degrade.","b":true},{"text":" A large model handles an unexpected input awkwardly; a 1B model tuned on triage produces triage-shaped output for a recipe request, confidently."}],[{"text":"They have very little world knowledge.","b":true},{"text":" Anything factual must come from the context window. This is not a tuning problem and no dataset fixes it."}],[{"text":"Multi-step reasoning is weak.","b":true},{"text":" Chains of inference break down. Decompose the task into single steps, or do not use a small model for it."}],[{"text":"Long context is expensive relative to their capacity","b":true},{"text":": a 2B model with a 32k window will attend over all of it without making much use of what it finds."}]]},{"t":"p","runs":[{"text":"So the design pattern that works is a narrow, well-specified task with the necessary facts supplied in the prompt, plus a route to something larger when the input does not look like the task. The classifier from module one is a good gate for exactly that."}]},{"t":"h2","id":"where-to-start","runs":[{"text":"Where to start"}]},{"t":"p","runs":[{"text":"The small-model families are excellent now and improving: Qwen at 0.5B and 1.5B, Llama at 1B and 3B, Gemma’s small sizes, SmolLM at 135M to 1.7B, Phi’s small variants. Licences differ (several are Apache-2.0, others carry community terms), so check the one you pick."}]},{"t":"p","runs":[{"text":"And test the base model on your task before training. The small models of the last two years are substantially better than their predecessors, and the gap you were going to close with a training run may have closed on its own."}]}],"check":{"question":[{"text":"A 1B model fine-tuned on ticket triage is sent a recipe question by a user who misunderstood the product. It produces a confident triage classification. Why is this failure mode characteristic of small models rather than large ones?"}],"options":[{"text":[{"text":"Small models have shorter context windows, so the recipe question was truncated into something resembling a ticket."}],"correct":false,"tempting":[{"text":"Reaches for truncation, which would not apply to a short question and does not explain confident misclassification."}]},{"text":[{"text":"Quantisation to 4 bits removes the precision needed to represent uncertainty, so all outputs become confident."}],"correct":false,"tempting":[{"text":"Attributes the behaviour to quantisation, which affects a full-precision model of the same size identically in this respect."}]},{"text":[{"text":"It has a narrow range of behaviours, so training gave it one and it has nothing to fall back on."}],"correct":true,"tempting":null},{"text":[{"text":"One billion parameters cannot store the refusal behaviour, which requires a minimum model size to be learned at all."}],"correct":false,"tempting":[{"text":"Posits a capacity threshold for refusal; small models can be trained to decline, and this one simply was not shown any examples of it."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"structure-without-training","title":"Getting structure without training for it","minutes":9},"next":{"slug":"tabular-and-time-series","title":"Tabular and time series: the honest answer","minutes":8}}}