{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"gguf-and-llama-cpp","title":"GGUF, and the laptop that has no GPU","url":"https://addaly.com/learn/hugging-face/gguf-and-llama-cpp","position":23,"of":75,"minutes":9,"answer":"GGUF packs weights, tokenizer and template into one self-contained file that llama.cpp runs on CPU, Apple silicon or a phone with no CUDA and no Python.","keyIdea":"GGUF packs weights, tokenizer and template into one self-contained file that llama.cpp runs on CPU, Apple silicon or a phone with no CUDA and no Python, making `Q4_K_M` the default free path for anyone without a graphics card."},"body":{"format":"markdown","text":"## The format built for the machine you have\n\n`llama.cpp` is a C++ implementation of transformer inference with no Python and no CUDA requirement. It runs on an ordinary CPU, on Apple silicon through Metal, on an Android phone, and on a Raspberry Pi. GGUF is its file format, and between them they are the most important free path in this entire course, because they are what makes open models usable by people without a graphics card.\n\nA GGUF file is self-contained: weights, tokenizer, architecture metadata and the chat template in one file. There is no `config.json` to match, no tokenizer to mismatch, no virtual environment to break. You download one file and run it. For anyone who has lost a day to a CUDA version conflict, this is a substantial thing.\n\n## Reading the quantization names\n\nInside a GGUF repository you will find a dozen files of the same model at different quantizations:\n\n- `Q8_0`: 8-bit, about half the bf16 size, quality effectively identical.\n- `Q6_K`: very close to Q8 at less memory.\n- `Q5_K_M`: the quality-conscious default.\n- `Q4_K_M`: **the usual recommendation**; roughly a quarter of the bf16 size, small quality loss.\n- `Q3_K_M`, `Q2_K`: for when nothing else fits. Visible damage, especially on small models.\n\nThe `_K` means k-quants, which allocate more bits to the layers that need them rather than treating all weights alike. `_S`, `_M`, `_L` are small, medium and large variants within a level. `IQ` prefixes are a newer scheme that squeezes more quality out of very low bit widths at some speed cost.\n\nFor a first choice: take `Q4_K_M` if your RAM is tight, `Q5_K_M` or `Q6_K` if you have room. Files are named with their size, so the arithmetic is done for you.\n\n## Running it\n\nThe llama.cpp server, which gives you an OpenAI-compatible HTTP endpoint on your own machine:\n\n```bash\nllama-server -hf unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080\n```\n\nThat flag downloads directly from the Hub. Then anything that speaks the OpenAI API (including most chat front-ends and the `openai` Python package pointed at `http://localhost:8080/v1`) works against a model running entirely on your own hardware, with no key and no data leaving the machine.\n\nFrom Python, without the server:\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama.from_pretrained(\n    repo_id=\"unsloth/Qwen2.5-7B-Instruct-GGUF\",\n    filename=\"*Q4_K_M.gguf\",\n    n_ctx=4096,\n)\nprint(llm.create_chat_completion(\n    messages=[{\"role\": \"user\", \"content\": \"Explain GGUF in two sentences.\"}]\n)[\"choices\"][0][\"message\"][\"content\"])\n```\n\nOllama and LM Studio wrap the same engine with a friendlier surface and pull GGUF files from the Hub too. Ollama is free and open source; LM Studio is free to use with a graphical interface. All three are running the same inference underneath.\n\n## Speed\n\nOn a recent laptop CPU with eight cores, a 7B model at `Q4_K_M` generates roughly 5 to 12 tokens per second. On Apple silicon with Metal, a 7B is often 20 to 40 tokens per second on an M-series chip with enough memory. A 3B model roughly doubles those figures; a 1B model is comfortably interactive on almost anything, including a mid-range phone.\n\nFor comparison, reading speed is about 4 tokens per second. So 7 tokens per second is slower than a fast API and faster than you can read, which for a private assistant is often enough.\n\nPrompt processing is separate and usually faster per token than generation, but a 4,000-token document still takes real seconds before the first output token appears. This is the number that makes long-context work on CPU feel bad, and `--n-gpu-layers` on a machine with any GPU at all (even a modest integrated one) offloads part of the model and helps disproportionately.\n\n## What GGUF does not do\n\n- **Training.** GGUF is an inference format. Fine-tune in PyTorch, convert afterwards.\n- **Every architecture.** Support is added per model family, so a brand-new architecture may have no GGUF for days or weeks.\n- **Custom heads.** A classifier with a bespoke output layer is not what this path is for.\n\nAnd one caution: because anyone can convert and upload, quality varies. A conversion done before a bug fix in llama.cpp can be subtly wrong. Prefer conversions from accounts that maintain them (the model's own organisation where it publishes one, or well-known conversion accounts) and check the repository's discussion tab before trusting an old file.\n\n## Do this now\n\nDownload a 1B or 3B instruct model at `Q4_K_M`, under 2.5 GB, and run it on whatever machine you have, including a phone if that is what you have. Time ten generations. That number is your floor, and everything else in this course is a choice about whether to exceed it."},"check":{"question":"Why does a GGUF file avoid the tokenizer-and-weights mismatch failures that plague transformers loading?","options":[{"text":"llama.cpp validates the vocabulary size against the embedding matrix at load and refuses to start when they disagree.","correct":false,"tempting":"Names a check that would catch the error rather than prevent it, and is not what makes the format robust."},{"text":"GGUF repositories are restricted to official model organisations, so conversions cannot introduce mismatched components.","correct":false,"tempting":"Anyone can convert and upload GGUF files, which is exactly why conversion provenance matters."},{"text":"Tokenizer, template, weights and metadata all live in the one file, so there is nothing to mismatch.","correct":true},{"text":"The format stores text as raw bytes rather than token ids, so no vocabulary needs to be shared between the two halves.","correct":false,"tempting":"Invents a byte-level runtime; GGUF models tokenize exactly as their originals do."}]},"prev":{"slug":"quantisation-four-ways","title":"Quantization: what you buy and what you pay","url":"https://addaly.com/learn/hugging-face/quantisation-four-ways"},"next":{"slug":"onnx-optimum-and-the-browser","title":"ONNX, Optimum, and models that run in a browser tab","url":"https://addaly.com/learn/hugging-face/onnx-optimum-and-the-browser"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'GGUF, and the laptop that has no GPU', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/gguf-and-llama-cpp (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"GGUF, and the laptop that has no GPU","courseTitle":"Hugging Face, End to End","module":{"index":3,"slug":"running-it-on-what-you-have","title":"Running it on the hardware you actually have"},"answer":null,"keyIdea":"GGUF packs weights, tokenizer and template into one self-contained file that llama.cpp runs on CPU, Apple silicon or a phone with no CUDA and no Python, making `Q4_K_M` the default free path for anyone without a graphics card.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-format-built-for-the-machine-you-have","runs":[{"text":"The format built for the machine you have"}]},{"t":"p","runs":[{"text":"llama.cpp","code":true},{"text":" is a C++ implementation of transformer inference with no Python and no CUDA requirement. It runs on an ordinary CPU, on Apple silicon through Metal, on an Android phone, and on a Raspberry Pi. GGUF is its file format, and between them they are the most important free path in this entire course, because they are what makes open models usable by people without a graphics card."}]},{"t":"p","runs":[{"text":"A GGUF file is self-contained: weights, tokenizer, architecture metadata and the chat template in one file. There is no "},{"text":"config.json","code":true},{"text":" to match, no tokenizer to mismatch, no virtual environment to break. You download one file and run it. For anyone who has lost a day to a CUDA version conflict, this is a substantial thing."}]},{"t":"h2","id":"reading-the-quantization-names","runs":[{"text":"Reading the quantization names"}]},{"t":"p","runs":[{"text":"Inside a GGUF repository you will find a dozen files of the same model at different quantizations:"}]},{"t":"ul","items":[[{"text":"Q8_0","code":true},{"text":": 8-bit, about half the bf16 size, quality effectively identical."}],[{"text":"Q6_K","code":true},{"text":": very close to Q8 at less memory."}],[{"text":"Q5_K_M","code":true},{"text":": the quality-conscious default."}],[{"text":"Q4_K_M","code":true},{"text":": "},{"text":"the usual recommendation","b":true},{"text":"; roughly a quarter of the bf16 size, small quality loss."}],[{"text":"Q3_K_M","code":true},{"text":", "},{"text":"Q2_K","code":true},{"text":": for when nothing else fits. Visible damage, especially on small models."}]]},{"t":"p","runs":[{"text":"The "},{"text":"_K","code":true},{"text":" means k-quants, which allocate more bits to the layers that need them rather than treating all weights alike. "},{"text":"_S","code":true},{"text":", "},{"text":"_M","code":true},{"text":", "},{"text":"_L","code":true},{"text":" are small, medium and large variants within a level. "},{"text":"IQ","code":true},{"text":" prefixes are a newer scheme that squeezes more quality out of very low bit widths at some speed cost."}]},{"t":"p","runs":[{"text":"For a first choice: take "},{"text":"Q4_K_M","code":true},{"text":" if your RAM is tight, "},{"text":"Q5_K_M","code":true},{"text":" or "},{"text":"Q6_K","code":true},{"text":" if you have room. Files are named with their size, so the arithmetic is done for you."}]},{"t":"h2","id":"running-it","runs":[{"text":"Running it"}]},{"t":"p","runs":[{"text":"The llama.cpp server, which gives you an OpenAI-compatible HTTP endpoint on your own machine:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -hf unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080"},{"t":"p","runs":[{"text":"That flag downloads directly from the Hub. Then anything that speaks the OpenAI API (including most chat front-ends and the "},{"text":"openai","code":true},{"text":" Python package pointed at "},{"text":"http://localhost:8080/v1","code":true},{"text":") works against a model running entirely on your own hardware, with no key and no data leaving the machine."}]},{"t":"p","runs":[{"text":"From Python, without the server:"}]},{"t":"code","lang":"python","label":"Python","text":"from llama_cpp import Llama\n\nllm = Llama.from_pretrained(\n    repo_id=\"unsloth/Qwen2.5-7B-Instruct-GGUF\",\n    filename=\"*Q4_K_M.gguf\",\n    n_ctx=4096,\n)\nprint(llm.create_chat_completion(\n    messages=[{\"role\": \"user\", \"content\": \"Explain GGUF in two sentences.\"}]\n)[\"choices\"][0][\"message\"][\"content\"])"},{"t":"p","runs":[{"text":"Ollama and LM Studio wrap the same engine with a friendlier surface and pull GGUF files from the Hub too. Ollama is free and open source; LM Studio is free to use with a graphical interface. All three are running the same inference underneath."}]},{"t":"h2","id":"speed","runs":[{"text":"Speed"}]},{"t":"p","runs":[{"text":"On a recent laptop CPU with eight cores, a 7B model at "},{"text":"Q4_K_M","code":true},{"text":" generates roughly 5 to 12 tokens per second. On Apple silicon with Metal, a 7B is often 20 to 40 tokens per second on an M-series chip with enough memory. A 3B model roughly doubles those figures; a 1B model is comfortably interactive on almost anything, including a mid-range phone."}]},{"t":"p","runs":[{"text":"For comparison, reading speed is about 4 tokens per second. So 7 tokens per second is slower than a fast API and faster than you can read, which for a private assistant is often enough."}]},{"t":"p","runs":[{"text":"Prompt processing is separate and usually faster per token than generation, but a 4,000-token document still takes real seconds before the first output token appears. This is the number that makes long-context work on CPU feel bad, and "},{"text":"--n-gpu-layers","code":true},{"text":" on a machine with any GPU at all (even a modest integrated one) offloads part of the model and helps disproportionately."}]},{"t":"h2","id":"what-gguf-does-not-do","runs":[{"text":"What GGUF does not do"}]},{"t":"ul","items":[[{"text":"Training.","b":true},{"text":" GGUF is an inference format. Fine-tune in PyTorch, convert afterwards."}],[{"text":"Every architecture.","b":true},{"text":" Support is added per model family, so a brand-new architecture may have no GGUF for days or weeks."}],[{"text":"Custom heads.","b":true},{"text":" A classifier with a bespoke output layer is not what this path is for."}]]},{"t":"p","runs":[{"text":"And one caution: because anyone can convert and upload, quality varies. A conversion done before a bug fix in llama.cpp can be subtly wrong. Prefer conversions from accounts that maintain them (the model’s own organisation where it publishes one, or well-known conversion accounts) and check the repository’s discussion tab before trusting an old file."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Download a 1B or 3B instruct model at "},{"text":"Q4_K_M","code":true},{"text":", under 2.5 GB, and run it on whatever machine you have, including a phone if that is what you have. Time ten generations. That number is your floor, and everything else in this course is a choice about whether to exceed it."}]}],"check":{"question":[{"text":"Why does a GGUF file avoid the tokenizer-and-weights mismatch failures that plague transformers loading?"}],"options":[{"text":[{"text":"llama.cpp validates the vocabulary size against the embedding matrix at load and refuses to start when they disagree."}],"correct":false,"tempting":[{"text":"Names a check that would catch the error rather than prevent it, and is not what makes the format robust."}]},{"text":[{"text":"GGUF repositories are restricted to official model organisations, so conversions cannot introduce mismatched components."}],"correct":false,"tempting":[{"text":"Anyone can convert and upload GGUF files, which is exactly why conversion provenance matters."}]},{"text":[{"text":"Tokenizer, template, weights and metadata all live in the one file, so there is nothing to mismatch."}],"correct":true,"tempting":null},{"text":[{"text":"The format stores text as raw bytes rather than token ids, so no vocabulary needs to be shared between the two halves."}],"correct":false,"tempting":[{"text":"Invents a byte-level runtime; GGUF models tokenize exactly as their originals do."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"quantisation-four-ways","title":"Quantization: what you buy and what you pay","minutes":10},"next":{"slug":"onnx-optimum-and-the-browser","title":"ONNX, Optimum, and models that run in a browser tab","minutes":9}}}