A model on your own laptop or phone

Why this is possible at all

The first module told you a model is a file of numbers. The history module told you the frontier files are hundreds of gigabytes and need racks of data-centre chips. Between those two facts sits the thing this lesson is about: small open-weights models, made smaller still, that run on the machine you already own.

Two things make it work. The lesson on scaling explained the first: since 2022 the labs have trained small models on enormous amounts of text, so that an eight-billion-parameter model of 2024 does what a hundred-and-seventy-five-billion-parameter model did in 2020. The second is called quantisation, and it is simple to picture.

Shrinking the numbers

Each parameter is stored, in training, as a sixteen-bit number: precise to about four decimal places. It turns out the model barely needs that precision to run. Round each number to one of sixteen levels (four bits) and the output is almost indistinguishable, while the file is a quarter of the size. An eight-billion-parameter model is sixteen gigabytes at full precision and under five at four bits, which fits in the memory of a laptop with eight gigabytes and leaves room for the operating system.

Quality does drop, a little, and drops more sharply below four bits. It is the cheapest trade in the field: a few per cent of benchmark score for a machine that costs nothing per query and holds nothing but your own text.

The tools

Three free tools do nearly all of it, and they are built on the same open-source engine.

Ollama is a command-line tool for Windows, Mac and Linux. Install it, and one line downloads and runs a model:

ollama run gemma3:4b

Type, and it answers. Swap the name for any of a few hundred models in its library; llama3.2, qwen3, phi4 and mistral are the commonest starting points. A second line turns it into a local server that any application on your machine can talk to as if it were a cloud chatbot.

LM Studio does the same with a graphical interface: a search box, a download button, a chat window, and a readout of how much memory each model needs before you fetch it. It is the friendlier first door.

On a phone, open-source apps run models of one to four billion parameters directly on the handset. A one-billion-parameter model is about half a gigabyte at four bits and answers at reading speed on a mid-range phone from the last few years; a four-billion one needs a recent flagship. Google publishes an open-source gallery app for exactly this on Android.

What speed to expect

A model generates one token at a time, and on a laptop without a graphics card the limit is how fast the processor can stream the whole file from memory for every token. An eight-billion-parameter model at four bits produces around five to ten tokens a second on a recent laptop processor, which is slower than you read but faster than you type. Apple’s recent laptops, whose memory is shared with a capable graphics unit, run the same model at twenty to fifty tokens a second. A gaming card with eight gigabytes of memory runs it faster still. Anything larger than about fourteen billion parameters is a poor experience without a dedicated card.

What a small model is good for

Everything in the reliable column of the first lesson of this module: rewriting, summarising a document you paste, drafting, translating between major languages, explaining code, answering questions about text you supply. It is good enough at those that a great deal of business software now quietly does them on-device.

What it is worse at is everything that was already unreliable, made more so. A four-billion-parameter model has a fraction of the frontier’s memory of facts, so invention in specifics is more frequent and starts earlier. Its reasoning over long chains is weaker. Its context window is usually shorter. And in most Indian languages it is markedly weaker than in English, for the reasons the fourth module gave, though several Indian labs publish small open models tuned for Hindi and other languages that improve on it.

Why bother

Three reasons, each of which the earlier lessons set up. Nothing leaves the machine, so the entire previous-but-one lesson on where your words go does not apply. The model cannot change under you: the file you downloaded is the file you run, next year too. And it works with no connection and no bill, which for a great many people on a great many networks is not a small thing.

The frontier will always be somewhere else. But the year-old frontier, in your own hands, for nothing, is a new fact about the world, and it is one that the companies selling the frontier do not advertise.