{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"ollama-and-the-friendly-path","title":"Ollama and the friendly path","url":"https://addaly.com/learn/open-models/ollama-and-the-friendly-path","position":35,"of":71,"minutes":8,"answer":"Wrappers like Ollama choose your quantisation, sampling defaults and context length for you.","keyIdea":"Wrappers like Ollama choose your quantisation, sampling defaults and context length for you, and the historic 2048-token context default is the single most common cause of a local model appearing to ignore the start of a document."},"body":{"format":"markdown","text":"## Two commands to a running model\n\n```bash\ncurl -fsSL https://ollama.com/install.sh | sh   # or brew install ollama\nollama run gemma3:4b\n```\n\nThat is a working local model with a chat prompt, a downloaded and correctly templated file, and an HTTP API on port 11434. For a first hour it is unbeatable, and for many people it is the permanent answer.\n\nThe alternatives in the same category: **LM Studio**, a graphical application with a model browser and a server tab, free but not open source; **Jan**, open source and similar in shape; **GPT4All**, aimed at document chat on ordinary laptops. All of them wrap llama.cpp, so the underlying behaviour is what the previous lesson described.\n\n## The default that catches everybody\n\n**Ollama's default context length has historically been 2048 tokens**, regardless of what the model supports. Send a longer prompt and the beginning is silently dropped.\n\nThe symptom is bizarre and very common: a model that answers well about the end of your document and appears not to have read the beginning, or a chat that forgets what was said ten turns ago. People report this as the model being poor. It is a configuration default.\n\n```bash\n# per session\nollama run qwen3:8b\n>>> /set parameter num_ctx 32768\n\n# or permanently, with a Modelfile\ncat > Modelfile <<'EOF'\nFROM qwen3:8b\nPARAMETER num_ctx 32768\nPARAMETER temperature 0\nEOF\nollama create qwen3-long -f Modelfile\n```\n\nCheck what you actually have with `ollama show qwen3:8b`, which prints the parameters and the template.\n\n## What else the wrapper decides for you\n\n**Quantisation.** A tag such as `gemma3:4b` resolves to a specific quantisation, usually Q4_K_M. If you want Q6 or Q8 you must ask: `gemma3:4b-instruct-q8_0`. Run `ollama list` after pulling to see the size and confirm which you got.\n\n**Sampling defaults.** Temperature, top-p and repetition penalty are set by the model's Modelfile, not by you, unless you override them. If you are comparing two models and did not set these explicitly, you are partly comparing their defaults.\n\n**The chat template.** Handled for you, from the file's metadata, which is the main thing you are buying.\n\n**Model residency.** Models unload after a few minutes of inactivity, so the first request after a pause pays the whole load time again. `OLLAMA_KEEP_ALIVE=-1` keeps a model resident indefinitely, at the cost of the memory.\n\n## Using it as a server\n\n```bash\nexport OLLAMA_HOST=0.0.0.0:11434     # only on a network you trust\ncurl localhost:11434/v1/chat/completions -d '{\n  \"model\":\"qwen3:8b\",\n  \"messages\":[{\"role\":\"user\",\"content\":\"hello\"}],\n  \"temperature\":0\n}'\n```\n\nThe OpenAI-compatible endpoint means every library and editor plugin works against it unchanged. That compatibility is worth more than any feature, because it means your evaluation script does not care whether it is talking to Ollama, llama.cpp, vLLM or a rented endpoint.\n\nThere is no authentication. Do not expose that port beyond a trusted network; a later lesson covers what happens when people do.\n\n## Importing your own file\n\n```bash\ncat > Modelfile <<'EOF'\nFROM ./my-finetune-Q4_K_M.gguf\nTEMPLATE \"{{ .Prompt }}\"\nPARAMETER stop \"<|im_end|>\"\nEOF\nollama create my-finetune -f Modelfile\n```\n\nUseful after you have converted your own fine-tune, and the point at which the wrapper stops being a black box.\n\n## When to move on\n\nStay with the friendly path while you are one person on one machine. Move to llama.cpp directly when you need a flag it does not expose, and to vLLM when you need to serve concurrent users properly.\n\nThe one habit to carry across regardless: **record the context length, the quantisation and the sampling parameters** alongside any result you intend to act on. Most irreproducible local benchmarks are irreproducible because one of those three was a default nobody wrote down."},"check":{"question":"A user pastes a 6,000-word document into a local chat and the model answers only about the final section, seeming not to have read the rest. Nothing errors. What is the first thing to check?","options":[{"text":"The quantisation level, since a Q4 model may be dropping detail from long inputs.","correct":false,"tempting":"Quantisation degrades quality gradually; it does not make the model behave as though the earlier text was absent."},{"text":"Whether the runtime's context length is set well below the model's maximum, so the earlier tokens are being truncated silently.","correct":true},{"text":"The chat template, since a wrong template would drop the earlier turns.","correct":false,"tempting":"A wrong template garbles roles and stop behaviour rather than selectively discarding the first part of a single message."},{"text":"The system prompt, which may be instructing the model to summarise only the conclusion.","correct":false,"tempting":"Worth ruling out, but it would not produce this pattern with no system prompt set, which is the default here."}]},"prev":{"slug":"llama-cpp-in-practice","title":"llama.cpp, end to end","url":"https://addaly.com/learn/open-models/llama-cpp-in-practice"},"next":{"slug":"predicting-tokens-per-second","title":"Predicting speed before you download","url":"https://addaly.com/learn/open-models/predicting-tokens-per-second"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ollama and the friendly path', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/ollama-and-the-friendly-path (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ollama and the friendly path","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Wrappers like Ollama choose your quantisation, sampling defaults and context length for you, and the historic 2048-token context default is the single most common cause of a local model appearing to ignore the start of a document.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-commands-to-a-running-model","runs":[{"text":"Two commands to a running model"}]},{"t":"code","lang":"bash","label":"Bash","text":"curl -fsSL https://ollama.com/install.sh | sh   # or brew install ollama\nollama run gemma3:4b"},{"t":"p","runs":[{"text":"That is a working local model with a chat prompt, a downloaded and correctly templated file, and an HTTP API on port 11434. For a first hour it is unbeatable, and for many people it is the permanent answer."}]},{"t":"p","runs":[{"text":"The alternatives in the same category: "},{"text":"LM Studio","b":true},{"text":", a graphical application with a model browser and a server tab, free but not open source; "},{"text":"Jan","b":true},{"text":", open source and similar in shape; "},{"text":"GPT4All","b":true},{"text":", aimed at document chat on ordinary laptops. All of them wrap llama.cpp, so the underlying behaviour is what the previous lesson described."}]},{"t":"h2","id":"the-default-that-catches-everybody","runs":[{"text":"The default that catches everybody"}]},{"t":"p","runs":[{"text":"Ollama’s default context length has historically been 2048 tokens","b":true},{"text":", regardless of what the model supports. Send a longer prompt and the beginning is silently dropped."}]},{"t":"p","runs":[{"text":"The symptom is bizarre and very common: a model that answers well about the end of your document and appears not to have read the beginning, or a chat that forgets what was said ten turns ago. People report this as the model being poor. It is a configuration default."}]},{"t":"code","lang":"bash","label":"Bash","text":"# per session\nollama run qwen3:8b\n>>> /set parameter num_ctx 32768\n\n# or permanently, with a Modelfile\ncat > Modelfile <<'EOF'\nFROM qwen3:8b\nPARAMETER num_ctx 32768\nPARAMETER temperature 0\nEOF\nollama create qwen3-long -f Modelfile"},{"t":"p","runs":[{"text":"Check what you actually have with "},{"text":"ollama show qwen3:8b","code":true},{"text":", which prints the parameters and the template."}]},{"t":"h2","id":"what-else-the-wrapper-decides-for-you","runs":[{"text":"What else the wrapper decides for you"}]},{"t":"p","runs":[{"text":"Quantisation.","b":true},{"text":" A tag such as "},{"text":"gemma3:4b","code":true},{"text":" resolves to a specific quantisation, usually Q4_K_M. If you want Q6 or Q8 you must ask: "},{"text":"gemma3:4b-instruct-q8_0","code":true},{"text":". Run "},{"text":"ollama list","code":true},{"text":" after pulling to see the size and confirm which you got."}]},{"t":"p","runs":[{"text":"Sampling defaults.","b":true},{"text":" Temperature, top-p and repetition penalty are set by the model’s Modelfile, not by you, unless you override them. If you are comparing two models and did not set these explicitly, you are partly comparing their defaults."}]},{"t":"p","runs":[{"text":"The chat template.","b":true},{"text":" Handled for you, from the file’s metadata, which is the main thing you are buying."}]},{"t":"p","runs":[{"text":"Model residency.","b":true},{"text":" Models unload after a few minutes of inactivity, so the first request after a pause pays the whole load time again. "},{"text":"OLLAMA_KEEP_ALIVE=-1","code":true},{"text":" keeps a model resident indefinitely, at the cost of the memory."}]},{"t":"h2","id":"using-it-as-a-server","runs":[{"text":"Using it as a server"}]},{"t":"code","lang":"bash","label":"Bash","text":"export OLLAMA_HOST=0.0.0.0:11434     # only on a network you trust\ncurl localhost:11434/v1/chat/completions -d '{\n  \"model\":\"qwen3:8b\",\n  \"messages\":[{\"role\":\"user\",\"content\":\"hello\"}],\n  \"temperature\":0\n}'"},{"t":"p","runs":[{"text":"The OpenAI-compatible endpoint means every library and editor plugin works against it unchanged. That compatibility is worth more than any feature, because it means your evaluation script does not care whether it is talking to Ollama, llama.cpp, vLLM or a rented endpoint."}]},{"t":"p","runs":[{"text":"There is no authentication. Do not expose that port beyond a trusted network; a later lesson covers what happens when people do."}]},{"t":"h2","id":"importing-your-own-file","runs":[{"text":"Importing your own file"}]},{"t":"code","lang":"bash","label":"Bash","text":"cat > Modelfile <<'EOF'\nFROM ./my-finetune-Q4_K_M.gguf\nTEMPLATE \"{{ .Prompt }}\"\nPARAMETER stop \"<|im_end|>\"\nEOF\nollama create my-finetune -f Modelfile"},{"t":"p","runs":[{"text":"Useful after you have converted your own fine-tune, and the point at which the wrapper stops being a black box."}]},{"t":"h2","id":"when-to-move-on","runs":[{"text":"When to move on"}]},{"t":"p","runs":[{"text":"Stay with the friendly path while you are one person on one machine. Move to llama.cpp directly when you need a flag it does not expose, and to vLLM when you need to serve concurrent users properly."}]},{"t":"p","runs":[{"text":"The one habit to carry across regardless: "},{"text":"record the context length, the quantisation and the sampling parameters","b":true},{"text":" alongside any result you intend to act on. Most irreproducible local benchmarks are irreproducible because one of those three was a default nobody wrote down."}]}],"check":{"question":[{"text":"A user pastes a 6,000-word document into a local chat and the model answers only about the final section, seeming not to have read the rest. Nothing errors. What is the first thing to check?"}],"options":[{"text":[{"text":"The quantisation level, since a Q4 model may be dropping detail from long inputs."}],"correct":false,"tempting":[{"text":"Quantisation degrades quality gradually; it does not make the model behave as though the earlier text was absent."}]},{"text":[{"text":"Whether the runtime’s context length is set well below the model’s maximum, so the earlier tokens are being truncated silently."}],"correct":true,"tempting":null},{"text":[{"text":"The chat template, since a wrong template would drop the earlier turns."}],"correct":false,"tempting":[{"text":"A wrong template garbles roles and stop behaviour rather than selectively discarding the first part of a single message."}]},{"text":[{"text":"The system prompt, which may be instructing the model to summarise only the conclusion."}],"correct":false,"tempting":[{"text":"Worth ruling out, but it would not produce this pattern with no system prompt set, which is the default here."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"llama-cpp-in-practice","title":"llama.cpp, end to end","minutes":10},"next":{"slug":"predicting-tokens-per-second","title":"Predicting speed before you download","minutes":9}}}