Ollama and the friendly path

Two commands to a running model

curl -fsSL https://ollama.com/install.sh | sh   # or brew install ollama
ollama run gemma3:4b

That is a working local model with a chat prompt, a downloaded and correctly templated file, and an HTTP API on port 11434. For a first hour it is unbeatable, and for many people it is the permanent answer.

The alternatives in the same category: LM Studio, a graphical application with a model browser and a server tab, free but not open source; Jan, open source and similar in shape; GPT4All, aimed at document chat on ordinary laptops. All of them wrap llama.cpp, so the underlying behaviour is what the previous lesson described.

The default that catches everybody

Ollama’s default context length has historically been 2048 tokens, regardless of what the model supports. Send a longer prompt and the beginning is silently dropped.

The symptom is bizarre and very common: a model that answers well about the end of your document and appears not to have read the beginning, or a chat that forgets what was said ten turns ago. People report this as the model being poor. It is a configuration default.

# per session
ollama run qwen3:8b
>>> /set parameter num_ctx 32768

# or permanently, with a Modelfile
cat > Modelfile <<'EOF'
FROM qwen3:8b
PARAMETER num_ctx 32768
PARAMETER temperature 0
EOF
ollama create qwen3-long -f Modelfile

Check what you actually have with ollama show qwen3:8b, which prints the parameters and the template.

What else the wrapper decides for you

Quantisation. A tag such as gemma3:4b resolves to a specific quantisation, usually Q4_K_M. If you want Q6 or Q8 you must ask: gemma3:4b-instruct-q8_0. Run ollama list after pulling to see the size and confirm which you got.

Sampling defaults. Temperature, top-p and repetition penalty are set by the model’s Modelfile, not by you, unless you override them. If you are comparing two models and did not set these explicitly, you are partly comparing their defaults.

The chat template. Handled for you, from the file’s metadata, which is the main thing you are buying.

Model residency. Models unload after a few minutes of inactivity, so the first request after a pause pays the whole load time again. OLLAMA_KEEP_ALIVE=-1 keeps a model resident indefinitely, at the cost of the memory.

Using it as a server

export OLLAMA_HOST=0.0.0.0:11434     # only on a network you trust
curl localhost:11434/v1/chat/completions -d '{
  "model":"qwen3:8b",
  "messages":[{"role":"user","content":"hello"}],
  "temperature":0
}'

The OpenAI-compatible endpoint means every library and editor plugin works against it unchanged. That compatibility is worth more than any feature, because it means your evaluation script does not care whether it is talking to Ollama, llama.cpp, vLLM or a rented endpoint.

There is no authentication. Do not expose that port beyond a trusted network; a later lesson covers what happens when people do.

Importing your own file

cat > Modelfile <<'EOF'
FROM ./my-finetune-Q4_K_M.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER stop "<|im_end|>"
EOF
ollama create my-finetune -f Modelfile

Useful after you have converted your own fine-tune, and the point at which the wrapper stops being a black box.

When to move on

Stay with the friendly path while you are one person on one machine. Move to llama.cpp directly when you need a flag it does not expose, and to vLLM when you need to serve concurrent users properly.

The one habit to carry across regardless: record the context length, the quantisation and the sampling parameters alongside any result you intend to act on. Most irreproducible local benchmarks are irreproducible because one of those three was a default nobody wrote down.