{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"ollama-and-llama-cpp","title":"Ollama and llama.cpp","url":"https://addaly.com/learn/running-models-yourself/ollama-and-llama-cpp","position":28,"of":82,"minutes":8,"answer":"Ollama is llama.cpp plus a registry, a template and sane defaults: leave it only for a flag it does not expose.","keyIdea":"Ollama is llama.cpp plus a registry, a template and sane defaults: leave it only for a flag it does not expose."},"body":{"format":"markdown","text":"People argue about these as if they were competitors. They are not. Ollama is a wrapper around llama.cpp's inference core with a model registry bolted on, and knowing exactly what it adds tells you exactly when to stop using it.\n\n## What Ollama actually is\n\nA Go application that bundles the ggml/llama.cpp inference engine and adds five things:\n\n1. **A model registry.** `ollama run qwen3:8b` fetches a quant someone already chose for you.\n2. **The correct chat template**, baked into the model file. This is the big one. A large share of \"this model is stupid\" reports are a wrong or missing template: the model never saw the format it was trained on.\n3. **A server on port 11434**, with an OpenAI-compatible route at `/v1`.\n4. **Automatic load and unload**, with GPU detection and layer offload chosen for you.\n5. **Modelfiles**, so you can pin a system prompt and parameters as a named model.\n\nThose five are precisely the things that are fiddly to get right by hand. That is why Ollama is the correct answer for most people, and why recommending llama.cpp to a beginner is usually showing off rather than helping.\n\n```bash\nollama run qwen3:8b\nollama ps                        # what is loaded, and on GPU or CPU\nollama show qwen3:8b --modelfile # the template and parameters it is using\n```\n\n## The default that catches everyone\n\nOllama allocates a modest context window by default (4096 tokens in recent versions, 2048 in older ones) regardless of what the model supports. Send 12,000 tokens and the oldest are dropped. No error, no warning. The model appears to have amnesia.\n\nFix it per session, per model, or globally:\n\n```bash\n# in an interactive session\n/set parameter num_ctx 16384\n\n# or for the whole server\nOLLAMA_CONTEXT_LENGTH=16384 ollama serve\n```\n\nRemember lesson 2: doing this costs memory, and on a small card it can push you into partial offload.\n\n## Where Ollama stops being enough\n\n- **You need a quant it does not carry.** Partly solved: `ollama run hf.co/user/repo:Q5_K_M` pulls GGUF from Hugging Face directly, but you still have less control than picking a file.\n- **You need samplers it does not expose.** DRY and XTC (lesson 7) are not there.\n- **You need to see the exact prompt string.** Debugging a template problem through Ollama is guesswork.\n- **You need real concurrency.** `OLLAMA_NUM_PARALLEL` exists and works for a handful of users. It is not a serving stack; lesson 6 covers what is.\n- **You want speculative decoding, specific tensor splits across named GPUs, or distributed inference.** llama.cpp has these; Ollama exposes some or none.\n\n## Using llama.cpp directly\n\n```bash\nllama-server \\\n  -m ~/models/Qwen3-8B-Q4_K_M.gguf \\\n  -c 16384 \\\n  -ngl 99 \\\n  --flash-attn \\\n  --cache-type-k q8_0 --cache-type-v q8_0 \\\n  --host 127.0.0.1 --port 8080\n```\n\n`-ngl 99` means offload every layer to the GPU; lower it to keep some on the CPU when memory is tight. The server exposes `/v1/chat/completions` and a usable web UI at the root.\n\nAnd the tool worth knowing about before you form any opinion about speed:\n\n```bash\nllama-bench -m model.gguf -p 512 -n 128\n```\n\nThat reports prompt-processing and token-generation rates separately, on your machine, with your build. Every benchmark you read online was measured on hardware that is not yours.\n\n## The progression\n\nStart with Ollama. Move to llama.cpp when you hit a flag Ollama does not expose, or when you need to see the raw prompt. Move to vLLM or SGLang when you have concurrent users and the model fits entirely in VRAM.\n\nMost people never need to leave step one, and that is a fact about Ollama being good, not about them being unambitious."},"check":{"question":"You paste a 12,000-token document into Ollama and ask for a summary. The model card says it supports 128k context. The summary only reflects the last part of the document. What is the most likely cause?","options":[{"text":"The server allocated a small context window by default, so the earliest tokens were dropped before the model ever saw them.","correct":true},{"text":"The model's effective context is far shorter than advertised, so it ignores material in the middle.","correct":false,"tempting":"Degraded mid-context recall is real, but it produces patchy attention, not a clean cut that keeps exactly the tail."},{"text":"The 4-bit quant has damaged long-context recall, so earlier material was represented too poorly to use.","correct":false,"tempting":"Quantisation does hurt long-range recall, but 12k is not long and the symptom here is precise truncation rather than fuzziness."},{"text":"You pulled the standard build and need the long-context variant of the same model.","correct":false,"tempting":"Treats context length as baked into the weights file rather than allocated by the server at load time."}]},"prev":{"slug":"pruning-distillation-and-other-shrinking","title":"The other ways to make a model smaller","url":"https://addaly.com/learn/running-models-yourself/pruning-distillation-and-other-shrinking"},"next":{"slug":"interfaces","title":"Interfaces: what each one is for","url":"https://addaly.com/learn/running-models-yourself/interfaces"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ollama and llama.cpp', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/ollama-and-llama-cpp (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ollama and llama.cpp","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"Ollama is llama.cpp plus a registry, a template and sane defaults: leave it only for a flag it does not expose.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"p","runs":[{"text":"People argue about these as if they were competitors. They are not. Ollama is a wrapper around llama.cpp’s inference core with a model registry bolted on, and knowing exactly what it adds tells you exactly when to stop using it."}],"lede":true},{"t":"h2","id":"what-ollama-actually-is","runs":[{"text":"What Ollama actually is"}]},{"t":"p","runs":[{"text":"A Go application that bundles the ggml/llama.cpp inference engine and adds five things:"}]},{"t":"ol","start":1,"items":[[{"text":"A model registry.","b":true},{"text":" "},{"text":"ollama run qwen3:8b","code":true},{"text":" fetches a quant someone already chose for you."}],[{"text":"The correct chat template","b":true},{"text":", baked into the model file. This is the big one. A large share of “this model is stupid” reports are a wrong or missing template: the model never saw the format it was trained on."}],[{"text":"A server on port 11434","b":true},{"text":", with an OpenAI-compatible route at "},{"text":"/v1","code":true},{"text":"."}],[{"text":"Automatic load and unload","b":true},{"text":", with GPU detection and layer offload chosen for you."}],[{"text":"Modelfiles","b":true},{"text":", so you can pin a system prompt and parameters as a named model."}]]},{"t":"p","runs":[{"text":"Those five are precisely the things that are fiddly to get right by hand. That is why Ollama is the correct answer for most people, and why recommending llama.cpp to a beginner is usually showing off rather than helping."}]},{"t":"code","lang":"bash","label":"Bash","text":"ollama run qwen3:8b\nollama ps                        # what is loaded, and on GPU or CPU\nollama show qwen3:8b --modelfile # the template and parameters it is using"},{"t":"h2","id":"the-default-that-catches-everyone","runs":[{"text":"The default that catches everyone"}]},{"t":"p","runs":[{"text":"Ollama allocates a modest context window by default (4096 tokens in recent versions, 2048 in older ones) regardless of what the model supports. Send 12,000 tokens and the oldest are dropped. No error, no warning. The model appears to have amnesia."}]},{"t":"p","runs":[{"text":"Fix it per session, per model, or globally:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# in an interactive session\n/set parameter num_ctx 16384\n\n# or for the whole server\nOLLAMA_CONTEXT_LENGTH=16384 ollama serve"},{"t":"p","runs":[{"text":"Remember lesson 2: doing this costs memory, and on a small card it can push you into partial offload."}]},{"t":"h2","id":"where-ollama-stops-being-enough","runs":[{"text":"Where Ollama stops being enough"}]},{"t":"ul","items":[[{"text":"You need a quant it does not carry.","b":true},{"text":" Partly solved: "},{"text":"ollama run hf.co/user/repo:Q5_K_M","code":true},{"text":" pulls GGUF from Hugging Face directly, but you still have less control than picking a file."}],[{"text":"You need samplers it does not expose.","b":true},{"text":" DRY and XTC (lesson 7) are not there."}],[{"text":"You need to see the exact prompt string.","b":true},{"text":" Debugging a template problem through Ollama is guesswork."}],[{"text":"You need real concurrency.","b":true},{"text":" "},{"text":"OLLAMA_NUM_PARALLEL","code":true},{"text":" exists and works for a handful of users. It is not a serving stack; lesson 6 covers what is."}],[{"text":"You want speculative decoding, specific tensor splits across named GPUs, or distributed inference.","b":true},{"text":" llama.cpp has these; Ollama exposes some or none."}]]},{"t":"h2","id":"using-llama-cpp-directly","runs":[{"text":"Using llama.cpp directly"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server \\\n  -m ~/models/Qwen3-8B-Q4_K_M.gguf \\\n  -c 16384 \\\n  -ngl 99 \\\n  --flash-attn \\\n  --cache-type-k q8_0 --cache-type-v q8_0 \\\n  --host 127.0.0.1 --port 8080"},{"t":"p","runs":[{"text":"-ngl 99","code":true},{"text":" means offload every layer to the GPU; lower it to keep some on the CPU when memory is tight. The server exposes "},{"text":"/v1/chat/completions","code":true},{"text":" and a usable web UI at the root."}]},{"t":"p","runs":[{"text":"And the tool worth knowing about before you form any opinion about speed:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-bench -m model.gguf -p 512 -n 128"},{"t":"p","runs":[{"text":"That reports prompt-processing and token-generation rates separately, on your machine, with your build. Every benchmark you read online was measured on hardware that is not yours."}]},{"t":"h2","id":"the-progression","runs":[{"text":"The progression"}]},{"t":"p","runs":[{"text":"Start with Ollama. Move to llama.cpp when you hit a flag Ollama does not expose, or when you need to see the raw prompt. Move to vLLM or SGLang when you have concurrent users and the model fits entirely in VRAM."}]},{"t":"p","runs":[{"text":"Most people never need to leave step one, and that is a fact about Ollama being good, not about them being unambitious."}]}],"check":{"question":[{"text":"You paste a 12,000-token document into Ollama and ask for a summary. The model card says it supports 128k context. The summary only reflects the last part of the document. What is the most likely cause?"}],"options":[{"text":[{"text":"The server allocated a small context window by default, so the earliest tokens were dropped before the model ever saw them."}],"correct":true,"tempting":null},{"text":[{"text":"The model’s effective context is far shorter than advertised, so it ignores material in the middle."}],"correct":false,"tempting":[{"text":"Degraded mid-context recall is real, but it produces patchy attention, not a clean cut that keeps exactly the tail."}]},{"text":[{"text":"The 4-bit quant has damaged long-context recall, so earlier material was represented too poorly to use."}],"correct":false,"tempting":[{"text":"Quantisation does hurt long-range recall, but 12k is not long and the symptom here is precise truncation rather than fuzziness."}]},{"text":[{"text":"You pulled the standard build and need the long-context variant of the same model."}],"correct":false,"tempting":[{"text":"Treats context length as baked into the weights file rather than allocated by the server at load time."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"pruning-distillation-and-other-shrinking","title":"The other ways to make a model smaller","minutes":8},"next":{"slug":"interfaces","title":"Interfaces: what each one is for","minutes":6}}}