The chat template is a contract

The most common self-inflicted wound

A team tries an open model, finds it rambles, ignores the system prompt and never stops, and concludes the model is poor. Nine times in ten the model is fine and the conversation was formatted wrongly. This lesson is how to check in thirty seconds.

An instruct model was trained on an exact byte sequence: special tokens marking where each role starts and ends, a specific ordering, a specific token that means the assistant should now speak. Deviate and you are sampling from a distribution the model was never trained on.

Where the template lives

Inside tokenizer_config.json, in a field called chat_template, written in Jinja. It ships with the model, which means the model tells you its own format and you never have to guess:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
print(tok.chat_template[:400])          # the source
print(tok.apply_chat_template(
    [{"role": "system", "content": "You are terse."},
     {"role": "user",   "content": "Summarise TCP in two lines."}],
    tokenize=False, add_generation_prompt=True))

Print that once for every model you use. It takes ten seconds and it removes an entire category of mystery.

For a Llama 3 model the output looks like this, and every one of those markers matters:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are terse.<|eot_id|><|start_header_id|>user<|end_header_id|>

Summarise TCP in two lines.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The four things that go wrong

Forgetting add_generation_prompt=True. Without it the string ends after the user’s turn and the model has not been told it is the assistant’s turn. It will often continue by writing another user message. This is the single most frequent cause of a model that talks to itself.

Double beginning-of-sequence tokens. The template already inserts <|begin_of_text|>. If you then call tok(formatted_string) with the default add_special_tokens=True, the tokenizer adds another. Two of them at the start measurably degrades output on some models and produces no error at all. Either use apply_chat_template(tokenize=True) and let it do both jobs, or pass add_special_tokens=False when tokenising a string the template already built.

Assuming a system role exists. Gemma’s template has no system role. Passing one either raises an exception or is silently folded into the first user turn depending on the version. Several other models accept a system message but were tuned with almost none, and follow it weakly. Read the template: if system does not appear in the Jinja source, the model does not have one.

Hand-writing the format from a blog post. Formats change between versions of the same family. Llama 2 used [INST] markers; Llama 3 uses header tokens; they share nothing. The template in the repository is correct by construction; a remembered format is not.

Why the chat endpoint is the right default

When you serve through llama.cpp’s server, vLLM or Ollama and call /v1/chat/completions, the server reads the template out of the model’s own metadata and applies it. That is a strong reason to use the chat endpoint rather than raw completions even when you are running locally and it feels like an unnecessary layer.

The exception is when you deliberately want no template: few-shot classification against a base model, perplexity measurement, or fill-in-the-middle code completion. Then use the completions endpoint and construct the string yourself, knowingly.

Tool calling lives here too

Models that support function calling encode tools in the template, usually as a JSON block in a system turn or a dedicated tools section, with a specific token or format for the model’s call. apply_chat_template accepts a tools= argument and will format them the way the model expects.

This is why tool calling that works on one model breaks on another with identical code: the format is per model, and a serving layer that does not implement that model’s variant will produce calls the model was not trained to emit.

The thirty-second diagnostic

When output is bad, before changing anything else:

1. Print the exact string being sent, including special tokens.
2. Compare it to apply_chat_template output for the same messages.
3. Check for a duplicated start token at the beginning.
4. Check the string ends with the assistant header, not with the user’s turn.

If all four are right, the problem is somewhere else and you have spent thirty seconds ruling out the likeliest cause.