What actually runs on eight gigabytes

The constraint most courses ignore

A large share of the people reading this are on a laptop with 8 GB of RAM and integrated graphics, or on a phone. The honest answer is not “you cannot do this”. It is that a specific and useful set of things runs well, and knowing which is more valuable than aspiring to the rest.

What fits, with numbers

On 8 GB of system RAM with no usable GPU, budget about 5 GB for models after the operating system and a browser:

• Embedding models: all-MiniLM-L6-v2 is 90 MB and encodes hundreds of sentences a second on a CPU. bge-small-en-v1.5 and the multilingual paraphrase-multilingual-MiniLM are similar. Semantic search over a few hundred thousand documents is entirely practical here.
• Encoder classifiers: DistilBERT-sized models, 250 MB, tens of items per second. Sentiment, intent, spam, topic, named entities.
• Speech recognition: whisper-tiny (75 MB) and whisper-base (145 MB) transcribe faster than real time on a CPU. whisper-small (480 MB) is slower than real time but usable for files.
• Small generative models at 4-bit: a 1B model is about 0.8 GB and runs at a readable speed; a 3B model is about 2 GB and is usable; a 7B model at Q4_K_M is about 4.4 GB and will run, slowly, if little else is open.
• Image models: background removal, OCR and small vision classifiers run fine. Image generation does not: even a distilled diffusion model wants several gigabytes and produces one image in minutes on a CPU.

What a phone does

A mid-range Android phone from the last few years runs, through llama.cpp-based apps or MLC, a 1B model at 4-bit at several tokens per second, and a 3B on a better handset. Whisper-tiny transcribes short clips. Browser-based transformers.js works on mobile browsers for small models, subject to the download.

The real constraints on a phone are thermal and battery: sustained generation heats the device and the system throttles it, so the tenth minute is much slower than the first. Design for bursts.

Choose a smaller model on purpose

The instinct is to reach for the largest model that fits. Often the better move is a task-specific small model:

• Sentiment: a 67M fine-tuned classifier beats an 8B general model on accuracy, runs 500 times faster, and costs nothing to serve.
• Entity extraction: a token classifier, same story.
• Search: an embedding model, not a chat model.

The general model is the right tool when the task is open-ended or changes weekly. When the task is fixed and narrow, a small specialised model is usually both better and cheaper, and the reflex to reach for a chat model for everything is one of the more expensive habits in the field.

Distillation, as a consumer of it

You do not need to train anything to benefit from distillation. distil-whisper/distil-large-v3 is roughly six times faster than whisper-large-v3 with a word error rate within about one percentage point on the data it was distilled for. distilbert is 40% smaller and 60% faster than BERT at about 97% of its accuracy on the standard benchmark suite. These are published, measured trades, and picking the distilled variant is often the single largest speed win available for no effort.

The caveat is in “for the data it was distilled for”: a distilled model can be relatively worse outside the distribution it was trained to imitate. Check on your own examples, as always.

Practical habits for a small machine

• Close the browser before running a large model. Modern browsers routinely hold 2 to 4 GB.
• Delete the cache you are not using. hf cache scan then hf cache delete. The blob store fills silently.
• Load once, reuse. Never create a pipeline inside a loop.
• Prefer ONNX or GGUF on CPU. Both are meaningfully faster than PyTorch eager on the same hardware.
• Use a Space for the heavy step. Encode your documents once on a free Space or a Colab session, download the vectors, and search them locally forever afterwards. Heavy work is often a one-off, and one-off work does not need local hardware.

The last resort that is not a defeat

When the model cannot run locally, calling a hosted API for that one step is a reasonable engineering decision, not a failure of principle. What matters is knowing which step needs it, so you rent for the ten per cent that requires it rather than the whole pipeline.

Do this now

Build a search over your own files with all-MiniLM-L6-v2: embed a few hundred documents, store the vectors in a NumPy array, and search by cosine similarity. It runs on any machine in this lesson, it needs no GPU, and it is a useful tool. Module six builds it properly.