llama.cpp, end to end

The tool that made local models real

llama.cpp is a C++ inference engine with no Python dependency, support for CPU, Nvidia, AMD, Apple Metal and Vulkan, and a single-file model format. Ollama, LM Studio, Jan and most desktop AI applications are wrappers around it. Learning the underlying tool takes an hour and gives you every dial the wrappers hide.

Installing

# macOS
brew install llama.cpp

# Debian or Ubuntu, prebuilt
# see the project releases; or build with the backend you need:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON        # Nvidia; use -DGGML_METAL=ON on a Mac
cmake --build build --config Release -j

The build flag is the important part. A CPU-only build on a machine with a GPU works and is several times slower, and it is a common quiet mistake.

Three binaries

llama-cli runs one prompt. llama-server exposes an OpenAI-compatible HTTP endpoint and a small web interface. llama-bench measures speed. You will use the server for almost everything.

llama-server   -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M   -c 8192   -ngl 99   --flash-attn   --port 8080

-hf downloads directly from Hugging Face with the quantisation after the colon, which removes a manual download step.

The flags that matter

-ngl N: number of layers to offload to the GPU. This is the single most important flag on a machine with a graphics card. -ngl 99 means all of them. If the model does not fit in VRAM, reduce it: the layers you offload run on the GPU, the rest on the CPU, and speed scales roughly with the fraction offloaded. Half the layers on a GPU is much better than none.

-c N: context length. It sets your KV cache allocation, so it is the flag that decides whether the model fits. Do not set it to the model’s maximum out of habit.

--flash-attn: a more memory-efficient attention implementation. Turn it on; there is rarely a reason not to.

--cache-type-k q8_0 --cache-type-v q8_0: quantise the KV cache, roughly halving its memory. Test the quality effect on long-context tasks.

-t N: CPU threads. The default is usually right; setting it above your physical core count reliably makes things slower.

--parallel N and -np N: number of concurrent sequences the server will handle. The context you set is divided among them, so -c 8192 --parallel 4 gives each slot 2048 tokens.

--mlock: lock the weights in RAM so the operating system cannot page them out. Worth it on a machine under memory pressure, and it will fail without the right permissions.

--no-mmap: turns off memory mapping. Generally leave mapping on; it lets a model start before the whole file is read.

Measuring before you tune

llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -ngl 99

This reports two numbers separately, and the separation is the point. Prompt processing is prefill: how fast it reads your input, measured in tokens per second and usually in the hundreds or thousands. Text generation is decode: how fast it produces output, usually one or two orders of magnitude slower. The next lesson explains why they differ so much.

Run this before and after any change. It is the difference between knowing that --flash-attn helped and believing it did.

Serving it

The server speaks the OpenAI API, so every client works:

curl localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role":"user","content":"Explain a hash join in three sentences."}],
  "temperature": 0, "max_tokens": 200
}'

The template comes from the GGUF metadata automatically. If output looks wrong, check what template it loaded: the startup log prints it, and a wrong template here is the most common cause of disappointing local results.

When to leave

llama.cpp is the right tool for one user, a handful of users, CPU inference, Apple hardware, and anything that must run offline on a laptop. It is not the right tool for serving thirty concurrent users on a datacentre GPU, where continuous batching and paged attention give several times the throughput. That is vLLM, two lessons from now.