Making a model small enough to be affordable

Why size becomes the problem

Offline, a model’s size does not matter. In production it decides three things: what hardware you need, what latency you can promise, and what the monthly bill is.

A model at 10,000 predictions per second costs real money per point of accuracy, and there is a size below which it runs on a phone and above which it does not. That threshold is often the difference between a product and a demo.

Quantisation

Store weights in fewer bits. A float32 weight becomes int8: four times smaller, and integer arithmetic is faster on most hardware.

Post-training quantisation converts a trained model directly. No retraining, a few lines, and accuracy typically falls by well under a point for int8. This is the first thing to try and it is often the only thing needed.

Quantisation-aware training simulates the reduced precision during training so the model adapts to it. Better accuracy at low bit widths, requires retraining.

For large language models, 4-bit quantisation via GPTQ or AWQ is what makes a 7-billion-parameter model run on a consumer GPU at all. The quality loss is real and modest, and the alternative is not running the model.

Distillation

Train a small “student” model to reproduce a large “teacher” model’s outputs: not the hard labels, but the full probability distribution.

The interesting part is why this works better than training the small model on the labels directly. A teacher that outputs [cat 0.7, lynx 0.25, car 0.05] is communicating that cats and lynxes are similar and neither resembles a car. The hard label cat contains none of that. Hinton and colleagues called it dark knowledge, and it means the student learns from a much richer signal than the original training data provided.

DistilBERT is 40% smaller and 60% faster than BERT while retaining about 97% of its performance on standard benchmarks. Those are typical numbers for a well-executed distillation.

Pruning

Remove weights that contribute little. Many trained networks have most of their weights near zero, and removing them changes the output hardly at all.

Unstructured pruning zeroes individual weights. Very high sparsity is achievable (90% is common) and it gives no speedup on standard hardware, because a sparse matrix with scattered zeros is still processed as a dense one.

Structured pruning removes whole channels, filters or attention heads. Lower achievable sparsity and a real speedup, because the resulting model is smaller.

The distinction matters and is frequently missed: unstructured pruning reduces the number of non-zero parameters, which is a different thing from reducing time or memory.

For tree models

The boosting equivalents, which are simpler and rarely discussed:

• Fewer trees. Early stopping already chose a number; check whether 60% of them give nearly the same score. Often they do.
• Fewer, deeper trees. 200 at depth 8 rather than 1,000 at depth 4, at similar accuracy and much faster traversal.
• Compile it. Treelite converts a tree ensemble to C and typically runs 2 to 10 times faster with identical predictions.
• Drop features. From module 6’s importance work. Fewer features means less to compute at serving time, which is usually the dominant cost anyway.

Deciding what to spend

The useful framing is cost per unit of accuracy, computed honestly.

A model at 0.847 AUC costs ₹40,000 a month to serve. A model at 0.839 costs ₹6,000. Is 0.008 of AUC worth ₹34,000 a month?

Sometimes obviously yes: in fraud, 0.008 might be crores of losses. Sometimes obviously no. The point is that this is an answerable question that most teams never ask, because the modelling and the infrastructure budget are held by different people.

Build the estimate. Predictions per month, cost per prediction, and the business value of the accuracy difference. It takes twenty minutes, it frequently changes the decision, and it is the kind of analysis that makes a machine learning person unusually valuable in a room of engineers.

Where to start

Measure first. Profile the serving path and find where the time actually goes, and recall from the serving lesson that it is usually feature retrieval rather than the model. Optimising a model that accounts for 12% of your latency caps your possible improvement at 12%, and people do this regularly because the model is the part they know how to change.