Hugging Face, End to End · Chapter 03

The memory arithmetic you can do in your head

2:05 · Narrated with a synthetic voice. Captions are available in the player.

Read the transcript

Whether something runs on your machine is one multiplication. Parameters times bytes per parameter. A seven billion parameter model is twenty eight gigabytes at full precision, fourteen at half, seven at eight bits and about four at four bits. You can do that before you click anything, and rule most models in or out from the name alone. But the weights are not the whole bill. The last bar here is the cache of keys and values that generation keeps for every token so far, and it grows with sequence length and with batch size. Eight conversations of four thousand tokens cost about as much again as a quantized model’s weights. That is why a job that loads perfectly runs out of memory halfway through a long answer. If you have no graphics card, the honest answer is not that you cannot do this. One file format packs the weights, the tokenizer and the chat template together, and a compact engine written in C++ runs that file on an ordinary processor, on Apple silicon, or on a phone, with no Python and no driver conflict. Four bits is the usual default and it costs a little quality that lands mostly on arithmetic and structured output. A small model exported to a portable graph runs inside a browser tab, which turns a static page into a private deployment that scales because the visitor’s device does the work. Free GPU hours exist too, on shared notebooks and shared pools, and every one of them is temporary. So write code that assumes the machine will vanish.

Sources and further reading (1)