Your disk fills up before your patience does

Two lines that quietly download two gigabytes

Running a model yourself is less dramatic than it sounds. Install one package and call one function:

pip install -U "transformers[torch]"

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english",
)
print(clf("The bus was late but the driver waited for me."))

That model is around 250 MB and classifies in a fraction of a second on any laptop CPU. No GPU, no configuration. The point of the example is not sentiment analysis, which is a solved and slightly boring task — it is that pipeline downloads the weights, the tokenizer and the config, caches them, and gives you a working function. Swap the model name and the same three lines do translation, transcription, captioning or entity extraction.

The cache, and the symlink that fools you

Everything you download goes to one shared cache, by default ~/.cache/huggingface/hub. Point it somewhere with room by setting one variable before you start:

export HF_HOME=/mnt/bigdisk/huggingface

Inside the cache, each repository has a blobs/ directory holding the actual bytes and a snapshots/<commit-hash>/ directory holding symbolic links with the human-readable filenames. Two revisions of the same model share every blob that did not change.

This structure produces one specific and very common disaster. You run out of disk, you go into the cache, you find a folder full of familiar filenames under snapshots/, you delete them, and nothing is freed. You deleted links. The blobs they pointed at are still there. Use the tool instead:

hf cache scan
hf cache delete

On older versions these are huggingface-cli scan-cache and huggingface-cli delete-cache. The scan prints every repository, its size and its revisions, which is also how you discover you have four copies of the same 13 GB model from four different tutorials.

The memory arithmetic

This is the whole of capacity planning, and it fits in five lines.

• 32-bit weights: 4 bytes per parameter.
• 16-bit: 2 bytes.
• 8-bit: 1 byte.
• 4-bit: roughly half a byte.

So a 7-billion-parameter model needs about 28, 14, 7 or 4 GB for the weights alone. Then add the key-value cache, which grows with how much context you feed it, and a gigabyte or two of framework overhead. A rule that holds: the weights plus about a quarter again.

The failure mode matters as much as the number. If a model does not fit in RAM, your machine does not usually raise a clean error — it starts swapping to disk, and the process appears to hang. People wait forty minutes for a first token believing it is slow when it is actually stuck. Check the size before you start, not while you wait.

What a machine with no GPU actually does

A great deal, once you stop trying to run the wrong things.

Small models run fine. Classification, embeddings, named entities, translation with a compact model, Whisper at tiny or base size. These are CPU-friendly because they are small, not because of any special trick.

Image generation is possible at low step counts. Most diffusion models need twenty to fifty passes through the network to make one image, which is what makes them hopeless on CPU. Distilled models cut that to one to four:

pip install -U diffusers torch pillow

from diffusers import AutoPipelineForText2Image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=torch.float32
).to("cpu")

img = pipe(
    "a steel tumbler of chai on a wooden table, morning light",
    num_inference_steps=2,
    guidance_scale=0.0,
).images[0]
img.save("chai.png")

About 2.5 GB to download, tens of seconds per image on a normal CPU, and the quality is well below SDXL. Note the licence — sd-turbo is non-commercial, exactly as lesson two warned. If the output is going anywhere near a paying client, this is not the model. /learn/ai-images covers the craft side.

Chat models run through llama.cpp, not through `transformers`. The GGUF format packs quantised weights into a single file, and the Hub is full of them from community quantisers. Q4_K_M is the usual balance of size against damage. Ollama and LM Studio both pull GGUF files straight from the Hub, and LM Studio has a graphical interface on Windows, macOS and Linux, so this path needs no Python at all. Rough expectations: with 8 GB of RAM a 3B model at 4-bit reads at a comfortable speed and a 7B is slow but usable; with 16 GB, 7B to 8B is comfortable. Apple silicon does unusually well because the CPU and GPU share memory. /learn/running-models-yourself goes into this properly.

A phone does not run these. What a phone runs is a browser pointed at a Space, or very small models compiled for the browser through transformers.js. That is not a limitation to work around; it is the reason lessons four and five exist.

Do this now

Run hf cache scan on whatever machine you have been experimenting on. Most people find several gigabytes they did not know about, and at least one model downloaded twice.