{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"pruning-distillation-and-other-shrinking","title":"The other ways to make a model smaller","url":"https://addaly.com/learn/running-models-yourself/pruning-distillation-and-other-shrinking","position":27,"of":82,"minutes":8,"answer":"Quantisation, pruning and distillation each remove a different kind of redundancy, and knowing which one a released model used predicts how it will fail: pruned models unevenly, distilled models confidently, quantised models on precision.","keyIdea":"Quantisation, pruning and distillation each remove a different kind of redundancy, and knowing which one a released model used predicts how it will fail: pruned models unevenly, distilled models confidently, quantised models on precision."},"body":{"format":"markdown","text":"## Quantisation is one lever of three\n\nA model can be made smaller by using fewer bits per weight, by having fewer weights, or by being a different model trained to imitate the first. These are not competitors; they compose, and the published small models you use every day are usually the result of all three.\n\n## Pruning: removing weights\n\n**Unstructured pruning** sets individual weights to zero: typically the smallest ones, or those a sensitivity measure marks as expendable. Fifty per cent of weights can often be removed with modest quality loss. The catch is that a sparse matrix with zeros scattered through it is no faster on ordinary hardware, because you still read and multiply the same shapes. You have saved storage and nothing else, unless the hardware supports a structured sparsity pattern such as two non-zeros in every group of four, which some NVIDIA generations accelerate directly.\n\n**Structured pruning** removes whole units (attention heads, feed-forward channels, or entire layers) so the resulting matrices are smaller and every engine runs them faster with no special support. This works, and the honest constraint is that it needs a recovery phase: after cutting, the model is retrained briefly on a modest corpus to heal the damage. Without that step the loss is large.\n\n**Depth pruning** is the blunt version and is instructive. Measure how much each layer changes its input, drop the layers that change it least, then heal. Middle layers in large models are often surprisingly redundant. Several published small models were produced exactly this way, and it is the cheapest structured method to try.\n\n## Distillation: training a smaller model to imitate a larger one\n\nThe student is trained not on hard labels but on the teacher's full output distribution, which carries far more information per example: not just \"the answer is B\" but \"B strongly, D plausibly, everything else negligible\". That extra signal is why a distilled model can beat the same architecture trained from scratch on the same data.\n\nTwo warnings.\n\n**The naming misleads.** A repository called `SomethingLarge-Distill-Qwen-7B` is a Qwen 7B fine-tuned on the large model's outputs. It is not the large model compressed. It inherits style, formatting and some reasoning habits; it does not inherit capability, and people are consistently disappointed because the name promised otherwise.\n\n**The legality is not always clear.** Training on outputs from a commercial API may breach that API's terms, whatever the copyright position on model outputs turns out to be. That question is unsettled and differs by jurisdiction; the contractual question usually is not, and it is the one that bites first.\n\n## How this composes in practice\n\nThe small models people run locally are typically: a large model distilled or trained on a curated corpus, sometimes structurally pruned, then quantisation-aware trained, then shipped at 4 bits. Each step compounds. This is why a current 4B is competitive with an 8B from two years earlier: it is not the same model made smaller, it is the product of a pipeline that did not exist then.\n\n## What you should and should not attempt\n\n**Worth doing yourself:** quantisation, always. It is an hour, it is reversible, and the tooling is mature.\n\n**Worth doing if you have a narrow task and some patience:** fine-tuning a small model on a larger one's outputs for your specific job. A 3B taught to do your one classification task can match a 32B at it, at a tenth of the cost. This is the most practical use of distillation for an individual, and it is covered properly later in this course.\n\n**Rarely worth doing yourself:** pruning. The recovery training is the expensive part, and unless you have a real corpus and real compute you will produce something worse than an off-the-shelf model of the target size. Use somebody else's pruned model instead; several are excellent.\n\n**Not a thing you can do:** convert a 70B into a good 8B. Every method above trades capability for size. The 8B you end up with will be an 8B, and a well-trained off-the-shelf 8B is probably better than the one you carved out of something larger.\n\n## The unifying idea\n\nAll of these are answers to one question: which parts of a trained network are not carrying their weight. Quantisation says the low-order bits. Pruning says whole units. Distillation says the whole architecture, and starts again with the large model as a teacher. Knowing which one a given released model used tells you where its weaknesses will be: a pruned model is uneven across capabilities, a distilled one is stylistically confident beyond its competence, and a quantised one fails on precision."},"check":{"question":"A team applies 50% unstructured magnitude pruning to a 7B model, confirms quality is only slightly reduced, and is surprised that inference speed is unchanged. What did they misunderstand?","options":[{"text":"Pruned weights are set to zero but the matrices keep their shape, so the same bytes are read and the same multiplies are performed unless the hardware supports a structured sparsity pattern.","correct":true},{"text":"Magnitude pruning removes the smallest weights, which contribute least to compute time, so the saving was always going to be negligible.","correct":false,"tempting":"Assumes compute cost varies with a weight's magnitude, which it does not; every element costs the same to multiply."},{"text":"Speed did not change because the KV cache, not the weights, dominates inference time at 7B.","correct":false,"tempting":"Reverses the dominant term; weight reads are gigabytes per token against a cache measured in kilobytes per token."},{"text":"The pruning needs a recovery fine-tune before the sparse kernels can be enabled, and quality was only preserved because recovery was skipped.","correct":false,"tempting":"Conflates the quality-recovery step with kernel support, which is a hardware and format question independent of retraining."}]},"prev":{"slug":"the-size-precision-frontier","title":"Bigger and crushed, or smaller and clean","url":"https://addaly.com/learn/running-models-yourself/the-size-precision-frontier"},"next":{"slug":"ollama-and-llama-cpp","title":"Ollama and llama.cpp","url":"https://addaly.com/learn/running-models-yourself/ollama-and-llama-cpp"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The other ways to make a model smaller', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/pruning-distillation-and-other-shrinking (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The other ways to make a model smaller","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"Quantisation, pruning and distillation each remove a different kind of redundancy, and knowing which one a released model used predicts how it will fail: pruned models unevenly, distilled models confidently, quantised models on precision.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"quantisation-is-one-lever-of-three","runs":[{"text":"Quantisation is one lever of three"}]},{"t":"p","runs":[{"text":"A model can be made smaller by using fewer bits per weight, by having fewer weights, or by being a different model trained to imitate the first. These are not competitors; they compose, and the published small models you use every day are usually the result of all three."}]},{"t":"h2","id":"pruning-removing-weights","runs":[{"text":"Pruning: removing weights"}]},{"t":"p","runs":[{"text":"Unstructured pruning","b":true},{"text":" sets individual weights to zero: typically the smallest ones, or those a sensitivity measure marks as expendable. Fifty per cent of weights can often be removed with modest quality loss. The catch is that a sparse matrix with zeros scattered through it is no faster on ordinary hardware, because you still read and multiply the same shapes. You have saved storage and nothing else, unless the hardware supports a structured sparsity pattern such as two non-zeros in every group of four, which some NVIDIA generations accelerate directly."}]},{"t":"p","runs":[{"text":"Structured pruning","b":true},{"text":" removes whole units (attention heads, feed-forward channels, or entire layers) so the resulting matrices are smaller and every engine runs them faster with no special support. This works, and the honest constraint is that it needs a recovery phase: after cutting, the model is retrained briefly on a modest corpus to heal the damage. Without that step the loss is large."}]},{"t":"p","runs":[{"text":"Depth pruning","b":true},{"text":" is the blunt version and is instructive. Measure how much each layer changes its input, drop the layers that change it least, then heal. Middle layers in large models are often surprisingly redundant. Several published small models were produced exactly this way, and it is the cheapest structured method to try."}]},{"t":"h2","id":"distillation-training-a-smaller-model-to-imitate-a-larger-on","runs":[{"text":"Distillation: training a smaller model to imitate a larger one"}]},{"t":"p","runs":[{"text":"The student is trained not on hard labels but on the teacher’s full output distribution, which carries far more information per example: not just “the answer is B” but “B strongly, D plausibly, everything else negligible”. That extra signal is why a distilled model can beat the same architecture trained from scratch on the same data."}]},{"t":"p","runs":[{"text":"Two warnings."}]},{"t":"p","runs":[{"text":"The naming misleads.","b":true},{"text":" A repository called "},{"text":"SomethingLarge-Distill-Qwen-7B","code":true},{"text":" is a Qwen 7B fine-tuned on the large model’s outputs. It is not the large model compressed. It inherits style, formatting and some reasoning habits; it does not inherit capability, and people are consistently disappointed because the name promised otherwise."}]},{"t":"p","runs":[{"text":"The legality is not always clear.","b":true},{"text":" Training on outputs from a commercial API may breach that API’s terms, whatever the copyright position on model outputs turns out to be. That question is unsettled and differs by jurisdiction; the contractual question usually is not, and it is the one that bites first."}]},{"t":"h2","id":"how-this-composes-in-practice","runs":[{"text":"How this composes in practice"}]},{"t":"p","runs":[{"text":"The small models people run locally are typically: a large model distilled or trained on a curated corpus, sometimes structurally pruned, then quantisation-aware trained, then shipped at 4 bits. Each step compounds. This is why a current 4B is competitive with an 8B from two years earlier: it is not the same model made smaller, it is the product of a pipeline that did not exist then."}]},{"t":"h2","id":"what-you-should-and-should-not-attempt","runs":[{"text":"What you should and should not attempt"}]},{"t":"p","runs":[{"text":"Worth doing yourself:","b":true},{"text":" quantisation, always. It is an hour, it is reversible, and the tooling is mature."}]},{"t":"p","runs":[{"text":"Worth doing if you have a narrow task and some patience:","b":true},{"text":" fine-tuning a small model on a larger one’s outputs for your specific job. A 3B taught to do your one classification task can match a 32B at it, at a tenth of the cost. This is the most practical use of distillation for an individual, and it is covered properly later in this course."}]},{"t":"p","runs":[{"text":"Rarely worth doing yourself:","b":true},{"text":" pruning. The recovery training is the expensive part, and unless you have a real corpus and real compute you will produce something worse than an off-the-shelf model of the target size. Use somebody else’s pruned model instead; several are excellent."}]},{"t":"p","runs":[{"text":"Not a thing you can do:","b":true},{"text":" convert a 70B into a good 8B. Every method above trades capability for size. The 8B you end up with will be an 8B, and a well-trained off-the-shelf 8B is probably better than the one you carved out of something larger."}]},{"t":"h2","id":"the-unifying-idea","runs":[{"text":"The unifying idea"}]},{"t":"p","runs":[{"text":"All of these are answers to one question: which parts of a trained network are not carrying their weight. Quantisation says the low-order bits. Pruning says whole units. Distillation says the whole architecture, and starts again with the large model as a teacher. Knowing which one a given released model used tells you where its weaknesses will be: a pruned model is uneven across capabilities, a distilled one is stylistically confident beyond its competence, and a quantised one fails on precision."}]}],"check":{"question":[{"text":"A team applies 50% unstructured magnitude pruning to a 7B model, confirms quality is only slightly reduced, and is surprised that inference speed is unchanged. What did they misunderstand?"}],"options":[{"text":[{"text":"Pruned weights are set to zero but the matrices keep their shape, so the same bytes are read and the same multiplies are performed unless the hardware supports a structured sparsity pattern."}],"correct":true,"tempting":null},{"text":[{"text":"Magnitude pruning removes the smallest weights, which contribute least to compute time, so the saving was always going to be negligible."}],"correct":false,"tempting":[{"text":"Assumes compute cost varies with a weight’s magnitude, which it does not; every element costs the same to multiply."}]},{"text":[{"text":"Speed did not change because the KV cache, not the weights, dominates inference time at 7B."}],"correct":false,"tempting":[{"text":"Reverses the dominant term; weight reads are gigabytes per token against a cache measured in kilobytes per token."}]},{"text":[{"text":"The pruning needs a recovery fine-tune before the sparse kernels can be enabled, and quality was only preserved because recovery was skipped."}],"correct":false,"tempting":[{"text":"Conflates the quality-recovery step with kernel support, which is a hardware and format question independent of retraining."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"the-size-precision-frontier","title":"Bigger and crushed, or smaller and clean","minutes":8},"next":{"slug":"ollama-and-llama-cpp","title":"Ollama and llama.cpp","minutes":8}}}