GGUF, and the laptop that has no GPU

The format built for the machine you have

llama.cpp is a C++ implementation of transformer inference with no Python and no CUDA requirement. It runs on an ordinary CPU, on Apple silicon through Metal, on an Android phone, and on a Raspberry Pi. GGUF is its file format, and between them they are the most important free path in this entire course, because they are what makes open models usable by people without a graphics card.

A GGUF file is self-contained: weights, tokenizer, architecture metadata and the chat template in one file. There is no config.json to match, no tokenizer to mismatch, no virtual environment to break. You download one file and run it. For anyone who has lost a day to a CUDA version conflict, this is a substantial thing.

Reading the quantization names

Inside a GGUF repository you will find a dozen files of the same model at different quantizations:

• Q8_0: 8-bit, about half the bf16 size, quality effectively identical.
• Q6_K: very close to Q8 at less memory.
• Q5_K_M: the quality-conscious default.
• Q4_K_M: the usual recommendation; roughly a quarter of the bf16 size, small quality loss.
• Q3_K_M, Q2_K: for when nothing else fits. Visible damage, especially on small models.

The _K means k-quants, which allocate more bits to the layers that need them rather than treating all weights alike. _S, _M, _L are small, medium and large variants within a level. IQ prefixes are a newer scheme that squeezes more quality out of very low bit widths at some speed cost.

For a first choice: take Q4_K_M if your RAM is tight, Q5_K_M or Q6_K if you have room. Files are named with their size, so the arithmetic is done for you.

Running it

The llama.cpp server, which gives you an OpenAI-compatible HTTP endpoint on your own machine:

llama-server -hf unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080

That flag downloads directly from the Hub. Then anything that speaks the OpenAI API (including most chat front-ends and the openai Python package pointed at http://localhost:8080/v1) works against a model running entirely on your own hardware, with no key and no data leaving the machine.

From Python, without the server:

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="unsloth/Qwen2.5-7B-Instruct-GGUF",
    filename="*Q4_K_M.gguf",
    n_ctx=4096,
)
print(llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain GGUF in two sentences."}]
)["choices"][0]["message"]["content"])

Ollama and LM Studio wrap the same engine with a friendlier surface and pull GGUF files from the Hub too. Ollama is free and open source; LM Studio is free to use with a graphical interface. All three are running the same inference underneath.

Speed

On a recent laptop CPU with eight cores, a 7B model at Q4_K_M generates roughly 5 to 12 tokens per second. On Apple silicon with Metal, a 7B is often 20 to 40 tokens per second on an M-series chip with enough memory. A 3B model roughly doubles those figures; a 1B model is comfortably interactive on almost anything, including a mid-range phone.

For comparison, reading speed is about 4 tokens per second. So 7 tokens per second is slower than a fast API and faster than you can read, which for a private assistant is often enough.

Prompt processing is separate and usually faster per token than generation, but a 4,000-token document still takes real seconds before the first output token appears. This is the number that makes long-context work on CPU feel bad, and --n-gpu-layers on a machine with any GPU at all (even a modest integrated one) offloads part of the model and helps disproportionately.

What GGUF does not do

• Training. GGUF is an inference format. Fine-tune in PyTorch, convert afterwards.
• Every architecture. Support is added per model family, so a brand-new architecture may have no GGUF for days or weeks.
• Custom heads. A classifier with a bespoke output layer is not what this path is for.

And one caution: because anyone can convert and upload, quality varies. A conversion done before a bug fix in llama.cpp can be subtly wrong. Prefer conversions from accounts that maintain them (the model’s own organisation where it publishes one, or well-known conversion accounts) and check the repository’s discussion tab before trusting an old file.

Do this now

Download a 1B or 3B instruct model at Q4_K_M, under 2.5 GB, and run it on whatever machine you have, including a phone if that is what you have. Time ten generations. That number is your floor, and everything else in this course is a choice about whether to exceed it.