Small models, where fine-tuning matters most

The effect is inverse to size

Fine-tuning a frontier-scale model on a narrow task gives a modest improvement, because the model was already close. Fine-tuning a 0.5B model on the same task can transform it, because the base model was barely adequate.

The mechanism: a small model has a weak prior about how to follow instructions and a narrow range of behaviours it can be prompted into. Prompting asks it to select a behaviour; training gives it one. On a task narrow enough, a 0.5B or 1.5B model tuned on a few thousand examples can match a very large model prompted, for that task, and only that task.

That qualification is the whole deal. You are trading breadth for a specific competence at a hundredth of the cost.

What runs where

In 4-bit quantisation, approximately:

• 0.5B: about 400 MB. Runs on a mid-range phone.
• 1.5B: about 1 GB. Comfortable on a phone, instant on a laptop.
• 3B: about 2 GB. Any modern laptop.
• 7B to 8B: about 5 GB. A laptop with 8 GB of RAM, at a few tokens a second on CPU and much faster with any GPU or on Apple silicon.

For training: below about 1.5B, prefer full fine-tuning over LoRA. It fits on a free T4 with 8-bit AdamW, it is simpler, and at that size the extra capacity to change is usually what you want rather than something to be constrained.

The serving stack, all free

• llama.cpp / GGUF: CPU and GPU, every desktop platform, and the format most other tools read. Ollama and LM Studio wrap it.
• ONNX Runtime: broad hardware support, good on Windows and on CPU.
• ExecuTorch for iOS and Android, MLC LLM for mobile GPUs, Core ML on Apple devices.

The path from a training run to a phone is: train the adapter, merge it, convert to GGUF, quantise, test. All of it free and all of it documented.

The real reason, which is not cost

Small on-device models are usually justified on price. That is the weakest argument, because a hosted small model is already cheap.

The strong reasons are the ones an API cannot offer at any price:

• The data does not leave the device. For a health app, a journal, a legal notes tool, this is the product rather than a feature.
• It works offline. On a train, in a rural clinic, on a factory floor.
• Latency has no network in it. 60 milliseconds to first token instead of 400.
• There is no per-request cost, so you can call the model for things that would never justify an API call: ranking every item in a list, checking every keystroke.
• No vendor can change or withdraw it. The model you shipped is the model that runs, for as long as the app exists.

The limitations, plainly

Small models fail differently from large ones, and worse:

• Off-distribution they collapse rather than degrade. A large model handles an unexpected input awkwardly; a 1B model tuned on triage produces triage-shaped output for a recipe request, confidently.
• They have very little world knowledge. Anything factual must come from the context window. This is not a tuning problem and no dataset fixes it.
• Multi-step reasoning is weak. Chains of inference break down. Decompose the task into single steps, or do not use a small model for it.
• Long context is expensive relative to their capacity: a 2B model with a 32k window will attend over all of it without making much use of what it finds.

So the design pattern that works is a narrow, well-specified task with the necessary facts supplied in the prompt, plus a route to something larger when the input does not look like the task. The classifier from module one is a good gate for exactly that.

Where to start

The small-model families are excellent now and improving: Qwen at 0.5B and 1.5B, Llama at 1B and 3B, Gemma’s small sizes, SmolLM at 135M to 1.7B, Phi’s small variants. Licences differ (several are Apache-2.0, others carry community terms), so check the one you pick.

And test the base model on your task before training. The small models of the last two years are substantially better than their predecessors, and the gap you were going to close with a training run may have closed on its own.