Chat templates: the most common silent failure

The model never sees your messages

A chat model does not receive a list of messages. It receives one string of tokens. Something has to convert [{"role": "user", "content": "hello"}] into that string, and the conversion has to match the format the model was fine-tuned on, exactly.

That conversion is the chat template. When it is wrong, the model is being addressed in a dialect it does not recognise. It still answers, models are robust, but noticeably worse: ignoring the system prompt, failing to stop, drifting out of character, or producing rambling text. This accounts for a large share of “this model is stupid” reports, and it is invisible unless you look.

What the string looks like

A typical modern format:

<|im_start|>system
You are a careful assistant.<|im_end|>
<|im_start|>user
What is the boiling point of water?<|im_end|>
<|im_start|>assistant

The markers are single tokens in the model’s vocabulary, learned during fine-tuning. The trailing <|im_start|>assistant is the cue to begin answering. Different families use different markers, and the differences are not cosmetic: a model trained on one set has no idea what another’s means.

Applying it correctly

The template is stored with the model, so let the tokeniser apply it:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

msgs = [{"role": "system", "content": "You are terse."},
        {"role": "user", "content": "Why is the sky blue?"}]

print(tok.apply_chat_template(msgs, tokenize=False,
                              add_generation_prompt=True))

Print it. Reading the actual string once teaches you more than any explanation, and it is the fastest diagnostic when something is off.

add_generation_prompt=True appends the assistant cue. Omit it and the model has not been told it is its turn, and will often continue the user’s message instead: a bug that looks like the model misunderstanding the question.

The three failures worth recognising

Double BOS. Many templates already emit a beginning-of-sequence token. If the tokeniser then adds another, the model starts with a sequence it has never seen. Symptoms are subtle: slightly worse quality, occasionally much worse. When calling a tokeniser directly, add_special_tokens=False on already-templated text is the fix.

A borrowed template. A community quant packaged with a template copied from a different model family. The markers are wrong, and the model is being spoken to in a foreign dialect. Dump the metadata and compare against the original repository’s tokenizer_config.json.

A system role the model does not have. Some models were trained without a system role. Passing one gets it silently folded into the first user message, or dropped. If your system prompt seems to have no effect, check whether the template actually places it anywhere.

Seeing what your server sends

If you use an OpenAI-compatible endpoint, the server applies the template for you and you never see the result. Two ways to look:

# llama-server logs the formatted prompt at higher verbosity
llama-server -m model.gguf --verbose

# or bypass chat entirely and send the raw string yourself
curl http://localhost:8080/v1/completions -H 'Content-Type: application/json' \
  -d '{"prompt":"<|im_start|>user\nhello<|im_end|>\n<|im_start|>assistant\n",
       "max_tokens":64}'

The second is the diagnostic that settles it. If the raw call behaves correctly and the chat call does not, the template is your problem.

Overriding a template

When the packaged template is wrong, replace it rather than working around it:

llama-server -m model.gguf --chat-template chatml
# or supply a Jinja file
llama-server -m model.gguf --chat-template-file ./correct.jinja

In Ollama the template lives in the Modelfile, and ollama show <model> --modelfile prints it. Copy it, correct it, and rebuild the model with ollama create.

Stop tokens

The other half of the same problem. The template’s end marker must be in the server’s stop list, or generation runs past the end of the answer and the model begins writing the next turn of the conversation itself: inventing a user question and answering it. If your model will not shut up, this is why, and the fix is a stop sequence rather than a shorter max_tokens.

Good engines read the stop tokens from the model’s metadata. When a model is new, or a quant was packaged carelessly, they are missing and you must supply them:

{"stop": ["<|im_end|>", "<|endoftext|>"]}

Check this on the first run of any new model. It takes one prompt and it saves a long, confused investigation.