{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"small-models-and-old-machines","title":"What actually runs on eight gigabytes","url":"https://addaly.com/learn/hugging-face/small-models-and-old-machines","position":26,"of":75,"minutes":9,"answer":"On 8 GB and no GPU, embeddings, encoder classifiers, small Whisper models and 1 to 3B generative models at 4-bit all run usefully.","keyIdea":"On 8 GB and no GPU, embeddings, encoder classifiers, small Whisper models and 1 to 3B generative models at 4-bit all run usefully, and for a fixed narrow task a small specialised model usually beats a large general one on accuracy, speed and cost at the same time."},"body":{"format":"markdown","text":"## The constraint most courses ignore\n\nA large share of the people reading this are on a laptop with 8 GB of RAM and integrated graphics, or on a phone. The honest answer is not \"you cannot do this\". It is that a specific and useful set of things runs well, and knowing which is more valuable than aspiring to the rest.\n\n## What fits, with numbers\n\nOn 8 GB of system RAM with no usable GPU, budget about 5 GB for models after the operating system and a browser:\n\n- **Embedding models**: `all-MiniLM-L6-v2` is 90 MB and encodes hundreds of sentences a second on a CPU. `bge-small-en-v1.5` and the multilingual `paraphrase-multilingual-MiniLM` are similar. Semantic search over a few hundred thousand documents is entirely practical here.\n- **Encoder classifiers**: DistilBERT-sized models, 250 MB, tens of items per second. Sentiment, intent, spam, topic, named entities.\n- **Speech recognition**: `whisper-tiny` (75 MB) and `whisper-base` (145 MB) transcribe faster than real time on a CPU. `whisper-small` (480 MB) is slower than real time but usable for files.\n- **Small generative models at 4-bit**: a 1B model is about 0.8 GB and runs at a readable speed; a 3B model is about 2 GB and is usable; a 7B model at `Q4_K_M` is about 4.4 GB and will run, slowly, if little else is open.\n- **Image models**: background removal, OCR and small vision classifiers run fine. Image *generation* does not: even a distilled diffusion model wants several gigabytes and produces one image in minutes on a CPU.\n\n## What a phone does\n\nA mid-range Android phone from the last few years runs, through llama.cpp-based apps or MLC, a 1B model at 4-bit at several tokens per second, and a 3B on a better handset. Whisper-tiny transcribes short clips. Browser-based transformers.js works on mobile browsers for small models, subject to the download.\n\nThe real constraints on a phone are thermal and battery: sustained generation heats the device and the system throttles it, so the tenth minute is much slower than the first. Design for bursts.\n\n## Choose a smaller model on purpose\n\nThe instinct is to reach for the largest model that fits. Often the better move is a **task-specific small model**:\n\n- Sentiment: a 67M fine-tuned classifier beats an 8B general model on accuracy, runs 500 times faster, and costs nothing to serve.\n- Entity extraction: a token classifier, same story.\n- Search: an embedding model, not a chat model.\n\nThe general model is the right tool when the task is open-ended or changes weekly. When the task is fixed and narrow, a small specialised model is usually both better and cheaper, and the reflex to reach for a chat model for everything is one of the more expensive habits in the field.\n\n## Distillation, as a consumer of it\n\nYou do not need to train anything to benefit from distillation. `distil-whisper/distil-large-v3` is roughly six times faster than `whisper-large-v3` with a word error rate within about one percentage point on the data it was distilled for. `distilbert` is 40% smaller and 60% faster than BERT at about 97% of its accuracy on the standard benchmark suite. These are published, measured trades, and picking the distilled variant is often the single largest speed win available for no effort.\n\nThe caveat is in \"for the data it was distilled for\": a distilled model can be relatively worse outside the distribution it was trained to imitate. Check on your own examples, as always.\n\n## Practical habits for a small machine\n\n- **Close the browser** before running a large model. Modern browsers routinely hold 2 to 4 GB.\n- **Delete the cache you are not using.** `hf cache scan` then `hf cache delete`. The blob store fills silently.\n- **Load once, reuse.** Never create a pipeline inside a loop.\n- **Prefer ONNX or GGUF on CPU.** Both are meaningfully faster than PyTorch eager on the same hardware.\n- **Use a Space for the heavy step.** Encode your documents once on a free Space or a Colab session, download the vectors, and search them locally forever afterwards. Heavy work is often a one-off, and one-off work does not need local hardware.\n\n## The last resort that is not a defeat\n\nWhen the model cannot run locally, calling a hosted API for that one step is a reasonable engineering decision, not a failure of principle. What matters is knowing which step needs it, so you rent for the ten per cent that requires it rather than the whole pipeline.\n\n## Do this now\n\nBuild a search over your own files with `all-MiniLM-L6-v2`: embed a few hundred documents, store the vectors in a NumPy array, and search by cosine similarity. It runs on any machine in this lesson, it needs no GPU, and it is a useful tool. Module six builds it properly."},"check":{"question":"A team needs sentiment labels on 200,000 short reviews using an ordinary laptop. Why is a 67M fine-tuned classifier usually the better choice than a 7B chat model at 4-bit?","options":[{"text":"Chat models cannot produce consistent labels because sampling introduces variation the classifier does not have.","correct":false,"tempting":"Sampling can be disabled, so determinism is not what separates the two here."},{"text":"Quantized models are unsuitable for classification because 4-bit damage concentrates in short outputs.","correct":false,"tempting":"Overextends a real point about quantization; short label outputs are not where the damage concentrates."},{"text":"The classifier was trained for this one task, so it is typically more accurate as well as hundreds of times faster.","correct":true},{"text":"The 7B model's context window is wasted on short reviews, so most of its compute goes to padding rather than content.","correct":false,"tempting":"Unused context costs nothing; compute scales with the tokens actually present."}]},"prev":{"slug":"free-gpus-and-their-limits","title":"Where the free GPUs actually are","url":"https://addaly.com/learn/hugging-face/free-gpus-and-their-limits"},"next":{"slug":"measuring-a-hardware-change","title":"Proving the change helped","url":"https://addaly.com/learn/hugging-face/measuring-a-hardware-change"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What actually runs on eight gigabytes', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/small-models-and-old-machines (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What actually runs on eight gigabytes","courseTitle":"Hugging Face, End to End","module":{"index":3,"slug":"running-it-on-what-you-have","title":"Running it on the hardware you actually have"},"answer":null,"keyIdea":"On 8 GB and no GPU, embeddings, encoder classifiers, small Whisper models and 1 to 3B generative models at 4-bit all run usefully, and for a fixed narrow task a small specialised model usually beats a large general one on accuracy, speed and cost at the same time.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-constraint-most-courses-ignore","runs":[{"text":"The constraint most courses ignore"}]},{"t":"p","runs":[{"text":"A large share of the people reading this are on a laptop with 8 GB of RAM and integrated graphics, or on a phone. The honest answer is not “you cannot do this”. It is that a specific and useful set of things runs well, and knowing which is more valuable than aspiring to the rest."}]},{"t":"h2","id":"what-fits-with-numbers","runs":[{"text":"What fits, with numbers"}]},{"t":"p","runs":[{"text":"On 8 GB of system RAM with no usable GPU, budget about 5 GB for models after the operating system and a browser:"}]},{"t":"ul","items":[[{"text":"Embedding models","b":true},{"text":": "},{"text":"all-MiniLM-L6-v2","code":true},{"text":" is 90 MB and encodes hundreds of sentences a second on a CPU. "},{"text":"bge-small-en-v1.5","code":true},{"text":" and the multilingual "},{"text":"paraphrase-multilingual-MiniLM","code":true},{"text":" are similar. Semantic search over a few hundred thousand documents is entirely practical here."}],[{"text":"Encoder classifiers","b":true},{"text":": DistilBERT-sized models, 250 MB, tens of items per second. Sentiment, intent, spam, topic, named entities."}],[{"text":"Speech recognition","b":true},{"text":": "},{"text":"whisper-tiny","code":true},{"text":" (75 MB) and "},{"text":"whisper-base","code":true},{"text":" (145 MB) transcribe faster than real time on a CPU. "},{"text":"whisper-small","code":true},{"text":" (480 MB) is slower than real time but usable for files."}],[{"text":"Small generative models at 4-bit","b":true},{"text":": a 1B model is about 0.8 GB and runs at a readable speed; a 3B model is about 2 GB and is usable; a 7B model at "},{"text":"Q4_K_M","code":true},{"text":" is about 4.4 GB and will run, slowly, if little else is open."}],[{"text":"Image models","b":true},{"text":": background removal, OCR and small vision classifiers run fine. Image "},{"text":"generation","i":true},{"text":" does not: even a distilled diffusion model wants several gigabytes and produces one image in minutes on a CPU."}]]},{"t":"h2","id":"what-a-phone-does","runs":[{"text":"What a phone does"}]},{"t":"p","runs":[{"text":"A mid-range Android phone from the last few years runs, through llama.cpp-based apps or MLC, a 1B model at 4-bit at several tokens per second, and a 3B on a better handset. Whisper-tiny transcribes short clips. Browser-based transformers.js works on mobile browsers for small models, subject to the download."}]},{"t":"p","runs":[{"text":"The real constraints on a phone are thermal and battery: sustained generation heats the device and the system throttles it, so the tenth minute is much slower than the first. Design for bursts."}]},{"t":"h2","id":"choose-a-smaller-model-on-purpose","runs":[{"text":"Choose a smaller model on purpose"}]},{"t":"p","runs":[{"text":"The instinct is to reach for the largest model that fits. Often the better move is a "},{"text":"task-specific small model","b":true},{"text":":"}]},{"t":"ul","items":[[{"text":"Sentiment: a 67M fine-tuned classifier beats an 8B general model on accuracy, runs 500 times faster, and costs nothing to serve."}],[{"text":"Entity extraction: a token classifier, same story."}],[{"text":"Search: an embedding model, not a chat model."}]]},{"t":"p","runs":[{"text":"The general model is the right tool when the task is open-ended or changes weekly. When the task is fixed and narrow, a small specialised model is usually both better and cheaper, and the reflex to reach for a chat model for everything is one of the more expensive habits in the field."}]},{"t":"h2","id":"distillation-as-a-consumer-of-it","runs":[{"text":"Distillation, as a consumer of it"}]},{"t":"p","runs":[{"text":"You do not need to train anything to benefit from distillation. "},{"text":"distil-whisper/distil-large-v3","code":true},{"text":" is roughly six times faster than "},{"text":"whisper-large-v3","code":true},{"text":" with a word error rate within about one percentage point on the data it was distilled for. "},{"text":"distilbert","code":true},{"text":" is 40% smaller and 60% faster than BERT at about 97% of its accuracy on the standard benchmark suite. These are published, measured trades, and picking the distilled variant is often the single largest speed win available for no effort."}]},{"t":"p","runs":[{"text":"The caveat is in “for the data it was distilled for”: a distilled model can be relatively worse outside the distribution it was trained to imitate. Check on your own examples, as always."}]},{"t":"h2","id":"practical-habits-for-a-small-machine","runs":[{"text":"Practical habits for a small machine"}]},{"t":"ul","items":[[{"text":"Close the browser","b":true},{"text":" before running a large model. Modern browsers routinely hold 2 to 4 GB."}],[{"text":"Delete the cache you are not using.","b":true},{"text":" "},{"text":"hf cache scan","code":true},{"text":" then "},{"text":"hf cache delete","code":true},{"text":". The blob store fills silently."}],[{"text":"Load once, reuse.","b":true},{"text":" Never create a pipeline inside a loop."}],[{"text":"Prefer ONNX or GGUF on CPU.","b":true},{"text":" Both are meaningfully faster than PyTorch eager on the same hardware."}],[{"text":"Use a Space for the heavy step.","b":true},{"text":" Encode your documents once on a free Space or a Colab session, download the vectors, and search them locally forever afterwards. Heavy work is often a one-off, and one-off work does not need local hardware."}]]},{"t":"h2","id":"the-last-resort-that-is-not-a-defeat","runs":[{"text":"The last resort that is not a defeat"}]},{"t":"p","runs":[{"text":"When the model cannot run locally, calling a hosted API for that one step is a reasonable engineering decision, not a failure of principle. What matters is knowing which step needs it, so you rent for the ten per cent that requires it rather than the whole pipeline."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Build a search over your own files with "},{"text":"all-MiniLM-L6-v2","code":true},{"text":": embed a few hundred documents, store the vectors in a NumPy array, and search by cosine similarity. It runs on any machine in this lesson, it needs no GPU, and it is a useful tool. Module six builds it properly."}]}],"check":{"question":[{"text":"A team needs sentiment labels on 200,000 short reviews using an ordinary laptop. Why is a 67M fine-tuned classifier usually the better choice than a 7B chat model at 4-bit?"}],"options":[{"text":[{"text":"Chat models cannot produce consistent labels because sampling introduces variation the classifier does not have."}],"correct":false,"tempting":[{"text":"Sampling can be disabled, so determinism is not what separates the two here."}]},{"text":[{"text":"Quantized models are unsuitable for classification because 4-bit damage concentrates in short outputs."}],"correct":false,"tempting":[{"text":"Overextends a real point about quantization; short label outputs are not where the damage concentrates."}]},{"text":[{"text":"The classifier was trained for this one task, so it is typically more accurate as well as hundreds of times faster."}],"correct":true,"tempting":null},{"text":[{"text":"The 7B model’s context window is wasted on short reviews, so most of its compute goes to padding rather than content."}],"correct":false,"tempting":[{"text":"Unused context costs nothing; compute scales with the tokens actually present."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"free-gpus-and-their-limits","title":"Where the free GPUs actually are","minutes":9},"next":{"slug":"measuring-a-hardware-change","title":"Proving the change helped","minutes":8}}}