{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"llama-cpp-in-practice","title":"llama.cpp, end to end","url":"https://addaly.com/learn/open-models/llama-cpp-in-practice","position":34,"of":71,"minutes":10,"answer":"llama.cpp is the engine under most desktop AI tools, and its important dials are the number of layers offloaded to the GPU, the context length that sets your cache allocation.","keyIdea":"llama.cpp is the engine under most desktop AI tools, and its important dials are the number of layers offloaded to the GPU, the context length that sets your cache allocation, and llama-bench reporting prefill and decode speed as two separate numbers."},"body":{"format":"markdown","text":"## The tool that made local models real\n\nllama.cpp is a C++ inference engine with no Python dependency, support for CPU, Nvidia, AMD, Apple Metal and Vulkan, and a single-file model format. Ollama, LM Studio, Jan and most desktop AI applications are wrappers around it. Learning the underlying tool takes an hour and gives you every dial the wrappers hide.\n\n## Installing\n\n```bash\n# macOS\nbrew install llama.cpp\n\n# Debian or Ubuntu, prebuilt\n# see the project releases; or build with the backend you need:\ngit clone https://github.com/ggml-org/llama.cpp && cd llama.cpp\ncmake -B build -DGGML_CUDA=ON        # Nvidia; use -DGGML_METAL=ON on a Mac\ncmake --build build --config Release -j\n```\n\nThe build flag is the important part. A CPU-only build on a machine with a GPU works and is several times slower, and it is a common quiet mistake.\n\n## Three binaries\n\n`llama-cli` runs one prompt. `llama-server` exposes an OpenAI-compatible HTTP endpoint and a small web interface. `llama-bench` measures speed. You will use the server for almost everything.\n\n```bash\nllama-server   -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M   -c 8192   -ngl 99   --flash-attn   --port 8080\n```\n\n`-hf` downloads directly from Hugging Face with the quantisation after the colon, which removes a manual download step.\n\n## The flags that matter\n\n**`-ngl N`**: number of layers to offload to the GPU. This is the single most important flag on a machine with a graphics card. `-ngl 99` means all of them. If the model does not fit in VRAM, reduce it: the layers you offload run on the GPU, the rest on the CPU, and speed scales roughly with the fraction offloaded. Half the layers on a GPU is much better than none.\n\n**`-c N`**: context length. It sets your KV cache allocation, so it is the flag that decides whether the model fits. Do not set it to the model's maximum out of habit.\n\n**`--flash-attn`**: a more memory-efficient attention implementation. Turn it on; there is rarely a reason not to.\n\n**`--cache-type-k q8_0 --cache-type-v q8_0`**: quantise the KV cache, roughly halving its memory. Test the quality effect on long-context tasks.\n\n**`-t N`**: CPU threads. The default is usually right; setting it above your physical core count reliably makes things slower.\n\n**`--parallel N`** and **`-np N`**: number of concurrent sequences the server will handle. The context you set is divided among them, so `-c 8192 --parallel 4` gives each slot 2048 tokens.\n\n**`--mlock`**: lock the weights in RAM so the operating system cannot page them out. Worth it on a machine under memory pressure, and it will fail without the right permissions.\n\n**`--no-mmap`**: turns off memory mapping. Generally leave mapping on; it lets a model start before the whole file is read.\n\n## Measuring before you tune\n\n```bash\nllama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -ngl 99\n```\n\nThis reports two numbers separately, and the separation is the point. **Prompt processing** is prefill: how fast it reads your input, measured in tokens per second and usually in the hundreds or thousands. **Text generation** is decode: how fast it produces output, usually one or two orders of magnitude slower. The next lesson explains why they differ so much.\n\nRun this before and after any change. It is the difference between knowing that `--flash-attn` helped and believing it did.\n\n## Serving it\n\nThe server speaks the OpenAI API, so every client works:\n\n```bash\ncurl localhost:8080/v1/chat/completions -H \"Content-Type: application/json\" -d '{\n  \"messages\": [{\"role\":\"user\",\"content\":\"Explain a hash join in three sentences.\"}],\n  \"temperature\": 0, \"max_tokens\": 200\n}'\n```\n\nThe template comes from the GGUF metadata automatically. If output looks wrong, check what template it loaded: the startup log prints it, and a wrong template here is the most common cause of disappointing local results.\n\n## When to leave\n\nllama.cpp is the right tool for one user, a handful of users, CPU inference, Apple hardware, and anything that must run offline on a laptop. It is not the right tool for serving thirty concurrent users on a datacentre GPU, where continuous batching and paged attention give several times the throughput. That is vLLM, two lessons from now."},"check":{"question":"A 7B model at Q4 needs about 4.9 GB but the GPU has only 4 GB free. What does -ngl let you do?","options":[{"text":"Nothing useful; the model must fit entirely in VRAM or run entirely on the CPU.","correct":false,"tempting":"Assumes all-or-nothing placement, which is exactly what layer-wise offload avoids."},{"text":"Offload as many layers as fit and run the rest on the CPU, giving speed between the two extremes roughly in proportion to the fraction offloaded.","correct":true},{"text":"Compress the model further at load time so that it fits in the available VRAM.","correct":false,"tempting":"Confuses offload with quantisation; -ngl moves layers, it does not change their precision."},{"text":"Swap layers in and out of VRAM per token, so the whole model effectively runs on the GPU.","correct":false,"tempting":"Per-token swapping over the bus would be far slower than simply running those layers on the CPU, which is why it is not what happens."}]},"prev":{"slug":"choosing-a-bit-width","title":"Choosing a bit width you can defend","url":"https://addaly.com/learn/open-models/choosing-a-bit-width"},"next":{"slug":"ollama-and-the-friendly-path","title":"Ollama and the friendly path","url":"https://addaly.com/learn/open-models/ollama-and-the-friendly-path"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'llama.cpp, end to end', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/llama-cpp-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"llama.cpp, end to end","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"llama.cpp is the engine under most desktop AI tools, and its important dials are the number of layers offloaded to the GPU, the context length that sets your cache allocation, and llama-bench reporting prefill and decode speed as two separate numbers.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-tool-that-made-local-models-real","runs":[{"text":"The tool that made local models real"}]},{"t":"p","runs":[{"text":"llama.cpp is a C++ inference engine with no Python dependency, support for CPU, Nvidia, AMD, Apple Metal and Vulkan, and a single-file model format. Ollama, LM Studio, Jan and most desktop AI applications are wrappers around it. Learning the underlying tool takes an hour and gives you every dial the wrappers hide."}]},{"t":"h2","id":"installing","runs":[{"text":"Installing"}]},{"t":"code","lang":"bash","label":"Bash","text":"# macOS\nbrew install llama.cpp\n\n# Debian or Ubuntu, prebuilt\n# see the project releases; or build with the backend you need:\ngit clone https://github.com/ggml-org/llama.cpp && cd llama.cpp\ncmake -B build -DGGML_CUDA=ON        # Nvidia; use -DGGML_METAL=ON on a Mac\ncmake --build build --config Release -j"},{"t":"p","runs":[{"text":"The build flag is the important part. A CPU-only build on a machine with a GPU works and is several times slower, and it is a common quiet mistake."}]},{"t":"h2","id":"three-binaries","runs":[{"text":"Three binaries"}]},{"t":"p","runs":[{"text":"llama-cli","code":true},{"text":" runs one prompt. "},{"text":"llama-server","code":true},{"text":" exposes an OpenAI-compatible HTTP endpoint and a small web interface. "},{"text":"llama-bench","code":true},{"text":" measures speed. You will use the server for almost everything."}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server   -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M   -c 8192   -ngl 99   --flash-attn   --port 8080"},{"t":"p","runs":[{"text":"-hf","code":true},{"text":" downloads directly from Hugging Face with the quantisation after the colon, which removes a manual download step."}]},{"t":"h2","id":"the-flags-that-matter","runs":[{"text":"The flags that matter"}]},{"t":"p","runs":[{"text":"-ngl N","b":true,"code":true},{"text":": number of layers to offload to the GPU. This is the single most important flag on a machine with a graphics card. "},{"text":"-ngl 99","code":true},{"text":" means all of them. If the model does not fit in VRAM, reduce it: the layers you offload run on the GPU, the rest on the CPU, and speed scales roughly with the fraction offloaded. Half the layers on a GPU is much better than none."}]},{"t":"p","runs":[{"text":"-c N","b":true,"code":true},{"text":": context length. It sets your KV cache allocation, so it is the flag that decides whether the model fits. Do not set it to the model’s maximum out of habit."}]},{"t":"p","runs":[{"text":"--flash-attn","b":true,"code":true},{"text":": a more memory-efficient attention implementation. Turn it on; there is rarely a reason not to."}]},{"t":"p","runs":[{"text":"--cache-type-k q8_0 --cache-type-v q8_0","b":true,"code":true},{"text":": quantise the KV cache, roughly halving its memory. Test the quality effect on long-context tasks."}]},{"t":"p","runs":[{"text":"-t N","b":true,"code":true},{"text":": CPU threads. The default is usually right; setting it above your physical core count reliably makes things slower."}]},{"t":"p","runs":[{"text":"--parallel N","b":true,"code":true},{"text":" and "},{"text":"-np N","b":true,"code":true},{"text":": number of concurrent sequences the server will handle. The context you set is divided among them, so "},{"text":"-c 8192 --parallel 4","code":true},{"text":" gives each slot 2048 tokens."}]},{"t":"p","runs":[{"text":"--mlock","b":true,"code":true},{"text":": lock the weights in RAM so the operating system cannot page them out. Worth it on a machine under memory pressure, and it will fail without the right permissions."}]},{"t":"p","runs":[{"text":"--no-mmap","b":true,"code":true},{"text":": turns off memory mapping. Generally leave mapping on; it lets a model start before the whole file is read."}]},{"t":"h2","id":"measuring-before-you-tune","runs":[{"text":"Measuring before you tune"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -ngl 99"},{"t":"p","runs":[{"text":"This reports two numbers separately, and the separation is the point. "},{"text":"Prompt processing","b":true},{"text":" is prefill: how fast it reads your input, measured in tokens per second and usually in the hundreds or thousands. "},{"text":"Text generation","b":true},{"text":" is decode: how fast it produces output, usually one or two orders of magnitude slower. The next lesson explains why they differ so much."}]},{"t":"p","runs":[{"text":"Run this before and after any change. It is the difference between knowing that "},{"text":"--flash-attn","code":true},{"text":" helped and believing it did."}]},{"t":"h2","id":"serving-it","runs":[{"text":"Serving it"}]},{"t":"p","runs":[{"text":"The server speaks the OpenAI API, so every client works:"}]},{"t":"code","lang":"bash","label":"Bash","text":"curl localhost:8080/v1/chat/completions -H \"Content-Type: application/json\" -d '{\n  \"messages\": [{\"role\":\"user\",\"content\":\"Explain a hash join in three sentences.\"}],\n  \"temperature\": 0, \"max_tokens\": 200\n}'"},{"t":"p","runs":[{"text":"The template comes from the GGUF metadata automatically. If output looks wrong, check what template it loaded: the startup log prints it, and a wrong template here is the most common cause of disappointing local results."}]},{"t":"h2","id":"when-to-leave","runs":[{"text":"When to leave"}]},{"t":"p","runs":[{"text":"llama.cpp is the right tool for one user, a handful of users, CPU inference, Apple hardware, and anything that must run offline on a laptop. It is not the right tool for serving thirty concurrent users on a datacentre GPU, where continuous batching and paged attention give several times the throughput. That is vLLM, two lessons from now."}]}],"check":{"question":[{"text":"A 7B model at Q4 needs about 4.9 GB but the GPU has only 4 GB free. What does -ngl let you do?"}],"options":[{"text":[{"text":"Nothing useful; the model must fit entirely in VRAM or run entirely on the CPU."}],"correct":false,"tempting":[{"text":"Assumes all-or-nothing placement, which is exactly what layer-wise offload avoids."}]},{"text":[{"text":"Offload as many layers as fit and run the rest on the CPU, giving speed between the two extremes roughly in proportion to the fraction offloaded."}],"correct":true,"tempting":null},{"text":[{"text":"Compress the model further at load time so that it fits in the available VRAM."}],"correct":false,"tempting":[{"text":"Confuses offload with quantisation; -ngl moves layers, it does not change their precision."}]},{"text":[{"text":"Swap layers in and out of VRAM per token, so the whole model effectively runs on the GPU."}],"correct":false,"tempting":[{"text":"Per-token swapping over the bus would be far slower than simply running those layers on the CPU, which is why it is not what happens."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"choosing-a-bit-width","title":"Choosing a bit width you can defend","minutes":8},"next":{"slug":"ollama-and-the-friendly-path","title":"Ollama and the friendly path","minutes":8}}}