{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"making-a-model-smaller","title":"Making a model small enough to be affordable","url":"https://addaly.com/learn/machine-learning-foundations/making-a-model-smaller","position":90,"of":92,"minutes":8,"answer":"Quantisation, distillation and pruning trade a small amount of accuracy for large reductions in size and latency; and unstructured pruning reduces parameter count without reducing runtime, which is the distinction most often missed.","keyIdea":"Quantisation, distillation and pruning trade a small amount of accuracy for large reductions in size and latency; and unstructured pruning reduces parameter count without reducing runtime, which is the distinction most often missed."},"body":{"format":"markdown","text":"## Why size becomes the problem\n\nOffline, a model's size does not matter. In production it decides three things: what hardware you need, what latency you can promise, and what the monthly bill is.\n\nA model at 10,000 predictions per second costs real money per point of accuracy, and there is a size below which it runs on a phone and above which it does not. That threshold is often the difference between a product and a demo.\n\n## Quantisation\n\nStore weights in fewer bits. A float32 weight becomes int8: four times smaller, and integer arithmetic is faster on most hardware.\n\n**Post-training quantisation** converts a trained model directly. No retraining, a few lines, and accuracy typically falls by well under a point for int8. This is the first thing to try and it is often the only thing needed.\n\n**Quantisation-aware training** simulates the reduced precision during training so the model adapts to it. Better accuracy at low bit widths, requires retraining.\n\nFor large language models, 4-bit quantisation via GPTQ or AWQ is what makes a 7-billion-parameter model run on a consumer GPU at all. The quality loss is real and modest, and the alternative is not running the model.\n\n## Distillation\n\nTrain a small \"student\" model to reproduce a large \"teacher\" model's outputs: not the hard labels, but the full probability distribution.\n\nThe interesting part is why this works better than training the small model on the labels directly. A teacher that outputs `[cat 0.7, lynx 0.25, car 0.05]` is communicating that cats and lynxes are similar and neither resembles a car. The hard label `cat` contains none of that. Hinton and colleagues called it dark knowledge, and it means the student learns from a much richer signal than the original training data provided.\n\nDistilBERT is 40% smaller and 60% faster than BERT while retaining about 97% of its performance on standard benchmarks. Those are typical numbers for a well-executed distillation.\n\n## Pruning\n\nRemove weights that contribute little. Many trained networks have most of their weights near zero, and removing them changes the output hardly at all.\n\n**Unstructured pruning** zeroes individual weights. Very high sparsity is achievable (90% is common) and it gives no speedup on standard hardware, because a sparse matrix with scattered zeros is still processed as a dense one.\n\n**Structured pruning** removes whole channels, filters or attention heads. Lower achievable sparsity and a real speedup, because the resulting model is smaller.\n\nThe distinction matters and is frequently missed: unstructured pruning reduces the number of non-zero parameters, which is a different thing from reducing time or memory.\n\n## For tree models\n\nThe boosting equivalents, which are simpler and rarely discussed:\n\n- **Fewer trees.** Early stopping already chose a number; check whether 60% of them give nearly the same score. Often they do.\n- **Fewer, deeper trees.** 200 at depth 8 rather than 1,000 at depth 4, at similar accuracy and much faster traversal.\n- **Compile it.** Treelite converts a tree ensemble to C and typically runs 2 to 10 times faster with identical predictions.\n- **Drop features.** From module 6's importance work. Fewer features means less to compute at serving time, which is usually the dominant cost anyway.\n\n## Deciding what to spend\n\nThe useful framing is cost per unit of accuracy, computed honestly.\n\nA model at 0.847 AUC costs ₹40,000 a month to serve. A model at 0.839 costs ₹6,000. Is 0.008 of AUC worth ₹34,000 a month?\n\nSometimes obviously yes: in fraud, 0.008 might be crores of losses. Sometimes obviously no. The point is that this is an answerable question that most teams never ask, because the modelling and the infrastructure budget are held by different people.\n\nBuild the estimate. Predictions per month, cost per prediction, and the business value of the accuracy difference. It takes twenty minutes, it frequently changes the decision, and it is the kind of analysis that makes a machine learning person unusually valuable in a room of engineers.\n\n## Where to start\n\nMeasure first. Profile the serving path and find where the time actually goes, and recall from the serving lesson that it is usually feature retrieval rather than the model. Optimising a model that accounts for 12% of your latency caps your possible improvement at 12%, and people do this regularly because the model is the part they know how to change."},"check":{"question":"A team prunes 90% of their network's weights using unstructured magnitude pruning and finds inference is no faster. Why?","options":[{"text":"The pruned weights were set to zero but not removed, and standard hardware processes a matrix with scattered zeros exactly as it processes a dense one.","correct":true},{"text":"90% sparsity is too aggressive, so the model compensates by requiring more forward passes.","correct":false,"tempting":"A network performs one forward pass regardless of sparsity; there is no compensating repetition."},{"text":"Pruning must be followed by quantisation to produce any speedup.","correct":false,"tempting":"Quantisation would help independently, but the absence of a speedup here is explained by the sparsity pattern rather than by precision."},{"text":"The speedup appears only after fine-tuning the pruned model to recover accuracy.","correct":false,"tempting":"Fine-tuning recovers accuracy lost to pruning; it does not change how the hardware processes the weight matrices."}]},"prev":{"slug":"automl-and-when-it-is-right","title":"AutoML: what it automates, and what it cannot","url":"https://addaly.com/learn/machine-learning-foundations/automl-and-when-it-is-right"},"next":{"slug":"feedback-loops","title":"When the model changes the data it will be trained on","url":"https://addaly.com/learn/machine-learning-foundations/feedback-loops"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Making a model small enough to be affordable', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/making-a-model-smaller (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Making a model small enough to be affordable","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"Quantisation, distillation and pruning trade a small amount of accuracy for large reductions in size and latency; and unstructured pruning reduces parameter count without reducing runtime, which is the distinction most often missed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-size-becomes-the-problem","runs":[{"text":"Why size becomes the problem"}]},{"t":"p","runs":[{"text":"Offline, a model’s size does not matter. In production it decides three things: what hardware you need, what latency you can promise, and what the monthly bill is."}]},{"t":"p","runs":[{"text":"A model at 10,000 predictions per second costs real money per point of accuracy, and there is a size below which it runs on a phone and above which it does not. That threshold is often the difference between a product and a demo."}]},{"t":"h2","id":"quantisation","runs":[{"text":"Quantisation"}]},{"t":"p","runs":[{"text":"Store weights in fewer bits. A float32 weight becomes int8: four times smaller, and integer arithmetic is faster on most hardware."}]},{"t":"p","runs":[{"text":"Post-training quantisation","b":true},{"text":" converts a trained model directly. No retraining, a few lines, and accuracy typically falls by well under a point for int8. This is the first thing to try and it is often the only thing needed."}]},{"t":"p","runs":[{"text":"Quantisation-aware training","b":true},{"text":" simulates the reduced precision during training so the model adapts to it. Better accuracy at low bit widths, requires retraining."}]},{"t":"p","runs":[{"text":"For large language models, 4-bit quantisation via GPTQ or AWQ is what makes a 7-billion-parameter model run on a consumer GPU at all. The quality loss is real and modest, and the alternative is not running the model."}]},{"t":"h2","id":"distillation","runs":[{"text":"Distillation"}]},{"t":"p","runs":[{"text":"Train a small “student” model to reproduce a large “teacher” model’s outputs: not the hard labels, but the full probability distribution."}]},{"t":"p","runs":[{"text":"The interesting part is why this works better than training the small model on the labels directly. A teacher that outputs "},{"text":"[cat 0.7, lynx 0.25, car 0.05]","code":true},{"text":" is communicating that cats and lynxes are similar and neither resembles a car. The hard label "},{"text":"cat","code":true},{"text":" contains none of that. Hinton and colleagues called it dark knowledge, and it means the student learns from a much richer signal than the original training data provided."}]},{"t":"p","runs":[{"text":"DistilBERT is 40% smaller and 60% faster than BERT while retaining about 97% of its performance on standard benchmarks. Those are typical numbers for a well-executed distillation."}]},{"t":"h2","id":"pruning","runs":[{"text":"Pruning"}]},{"t":"p","runs":[{"text":"Remove weights that contribute little. Many trained networks have most of their weights near zero, and removing them changes the output hardly at all."}]},{"t":"p","runs":[{"text":"Unstructured pruning","b":true},{"text":" zeroes individual weights. Very high sparsity is achievable (90% is common) and it gives no speedup on standard hardware, because a sparse matrix with scattered zeros is still processed as a dense one."}]},{"t":"p","runs":[{"text":"Structured pruning","b":true},{"text":" removes whole channels, filters or attention heads. Lower achievable sparsity and a real speedup, because the resulting model is smaller."}]},{"t":"p","runs":[{"text":"The distinction matters and is frequently missed: unstructured pruning reduces the number of non-zero parameters, which is a different thing from reducing time or memory."}]},{"t":"h2","id":"for-tree-models","runs":[{"text":"For tree models"}]},{"t":"p","runs":[{"text":"The boosting equivalents, which are simpler and rarely discussed:"}]},{"t":"ul","items":[[{"text":"Fewer trees.","b":true},{"text":" Early stopping already chose a number; check whether 60% of them give nearly the same score. Often they do."}],[{"text":"Fewer, deeper trees.","b":true},{"text":" 200 at depth 8 rather than 1,000 at depth 4, at similar accuracy and much faster traversal."}],[{"text":"Compile it.","b":true},{"text":" Treelite converts a tree ensemble to C and typically runs 2 to 10 times faster with identical predictions."}],[{"text":"Drop features.","b":true},{"text":" From module 6’s importance work. Fewer features means less to compute at serving time, which is usually the dominant cost anyway."}]]},{"t":"h2","id":"deciding-what-to-spend","runs":[{"text":"Deciding what to spend"}]},{"t":"p","runs":[{"text":"The useful framing is cost per unit of accuracy, computed honestly."}]},{"t":"p","runs":[{"text":"A model at 0.847 AUC costs ₹40,000 a month to serve. A model at 0.839 costs ₹6,000. Is 0.008 of AUC worth ₹34,000 a month?"}]},{"t":"p","runs":[{"text":"Sometimes obviously yes: in fraud, 0.008 might be crores of losses. Sometimes obviously no. The point is that this is an answerable question that most teams never ask, because the modelling and the infrastructure budget are held by different people."}]},{"t":"p","runs":[{"text":"Build the estimate. Predictions per month, cost per prediction, and the business value of the accuracy difference. It takes twenty minutes, it frequently changes the decision, and it is the kind of analysis that makes a machine learning person unusually valuable in a room of engineers."}]},{"t":"h2","id":"where-to-start","runs":[{"text":"Where to start"}]},{"t":"p","runs":[{"text":"Measure first. Profile the serving path and find where the time actually goes, and recall from the serving lesson that it is usually feature retrieval rather than the model. Optimising a model that accounts for 12% of your latency caps your possible improvement at 12%, and people do this regularly because the model is the part they know how to change."}]}],"check":{"question":[{"text":"A team prunes 90% of their network’s weights using unstructured magnitude pruning and finds inference is no faster. Why?"}],"options":[{"text":[{"text":"The pruned weights were set to zero but not removed, and standard hardware processes a matrix with scattered zeros exactly as it processes a dense one."}],"correct":true,"tempting":null},{"text":[{"text":"90% sparsity is too aggressive, so the model compensates by requiring more forward passes."}],"correct":false,"tempting":[{"text":"A network performs one forward pass regardless of sparsity; there is no compensating repetition."}]},{"text":[{"text":"Pruning must be followed by quantisation to produce any speedup."}],"correct":false,"tempting":[{"text":"Quantisation would help independently, but the absence of a speedup here is explained by the sparsity pattern rather than by precision."}]},{"text":[{"text":"The speedup appears only after fine-tuning the pruned model to recover accuracy."}],"correct":false,"tempting":[{"text":"Fine-tuning recovers accuracy lost to pruning; it does not change how the hardware processes the weight matrices."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"automl-and-when-it-is-right","title":"AutoML: what it automates, and what it cannot","minutes":8},"next":{"slug":"feedback-loops","title":"When the model changes the data it will be trained on","minutes":9}}}