Ollama and llama.cpp · Running Models Yourself

People argue about these as if they were competitors. They are not. Ollama is a wrapper around llama.cpp's inference core with a model registry bolted on, and knowing exactly what it adds tells you exactly when to stop using it.

What Ollama actually is

A Go application that bundles the ggml/llama.cpp inference engine and adds five things:

1A model registry. ollama run qwen3:8b fetches a quant someone already chose for you.
2The correct chat template, baked into the model file. This is the big one. A large share of "this model is stupid" reports are a wrong or missing template — the model never saw the format it was trained on.
3A server on port 11434, with an OpenAI-compatible route at /v1.
4Automatic load and unload, with GPU detection and layer offload chosen for you.
5Modelfiles, so you can pin a system prompt and parameters as a named model.

Those five are precisely the things that are fiddly to get right by hand. That is why Ollama is the correct answer for most people, and why recommending llama.cpp to a beginner is usually showing off rather than helping.

bash

ollama run qwen3:8b
ollama ps                        # what is loaded, and on GPU or CPU
ollama show qwen3:8b --modelfile # the template and parameters it is using

The default that catches everyone

Ollama allocates a modest context window by default — 4096 tokens in recent versions, 2048 in older ones — regardless of what the model supports. Send 12,000 tokens and the oldest are dropped. No error, no warning. The model appears to have amnesia.

Fix it per session, per model, or globally:

bash

# in an interactive session
/set parameter num_ctx 16384

# or for the whole server
OLLAMA_CONTEXT_LENGTH=16384 ollama serve

Remember lesson 2: doing this costs memory, and on a small card it can push you into partial offload.

Where Ollama stops being enough

You need a quant it does not carry. Partly solved — ollama run hf.co/user/repo:Q5_K_M pulls GGUF from Hugging Face directly — but you still have less control than picking a file.
You need samplers it does not expose. DRY and XTC (lesson 7) are not there.
You need to see the exact prompt string. Debugging a template problem through Ollama is guesswork.
You need real concurrency. OLLAMA_NUM_PARALLEL exists and works for a handful of users. It is not a serving stack; lesson 6 covers what is.
You want speculative decoding, specific tensor splits across named GPUs, or distributed inference. llama.cpp has these; Ollama exposes some or none.

Using llama.cpp directly

bash

llama-server \
  -m ~/models/Qwen3-8B-Q4_K_M.gguf \
  -c 16384 \
  -ngl 99 \
  --flash-attn \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 127.0.0.1 --port 8080

-ngl 99 means offload every layer to the GPU; lower it to keep some on the CPU when memory is tight. The server exposes /v1/chat/completions and a usable web UI at the root.

And the tool worth knowing about before you form any opinion about speed:

bash

llama-bench -m model.gguf -p 512 -n 128

That reports prompt-processing and token-generation rates separately, on your machine, with your build. Every benchmark you read online was measured on hardware that is not yours.

The progression

Start with Ollama. Move to llama.cpp when you hit a flag Ollama does not expose, or when you need to see the raw prompt. Move to vLLM or SGLang when you have concurrent users and the model fits entirely in VRAM.

Most people never need to leave step one, and that is a fact about Ollama being good, not about them being unambitious.

Before you move on

You paste a 12,000-token document into Ollama and ask for a summary. The model card says it supports 128k context. The summary only reflects the last part of the document. What is the most likely cause?

Pick the one you would defend. Nobody sees your answer.