The chat template is not optional

The most common quality bug in open models

Somebody downloads an instruct model, writes this, and concludes it is weak:

prompt = "User: Summarise this in two lines.\n\nAssistant:"

That is not how the model was trained. During instruction tuning it saw a specific string (specific role markers, specific special tokens, specific newlines) thousands of times. Give it something that looks similar but is not identical and you are outside the distribution it learned. Output gets vaguer, it ignores system instructions, it fails to stop, it starts inventing a second user turn.

The correct format is not a convention you can guess. It is stored in the repository, in tokenizer_config.json, as a Jinja string named chat_template.

Use it

messages = [
    {"role": "system", "content": "You reply in one sentence."},
    {"role": "user", "content": "Why is the sky blue?"},
]

text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
print(repr(text))

For a ChatML-style model that prints something like:

<|im_start|>system
You reply in one sentence.<|im_end|>
<|im_start|>user
Why is the sky blue?<|im_end|>
<|im_start|>assistant

Every model family differs. Llama 3 uses <|start_header_id|> markers, Mistral uses [INST] and [/INST], Gemma uses <start_of_turn>. Guessing wrong is invisible in the output shape and visible in the quality.

add_generation_prompt=True is the part people miss. It appends the opening marker for the assistant’s turn, so the model knows it is its turn to speak. Without it the model may produce another user message, because that is what would plausibly come next.

Print it once per model

tokenize=False exists so you can look. Do it once for every new model: it takes five seconds and tells you the system-role support, the turn markers, and the stop token:

print(tok.chat_template)          # the Jinja source
print(tok.eos_token, tok.eos_token_id)

Some models have no system role, and their template silently folds the system message into the first user turn, or drops it. That explains a whole class of “it ignores my instructions” reports.

In the pipeline

The text-generation pipeline applies the template for you when you pass a list of messages rather than a string:

pipe = pipeline("text-generation", model="Qwen/Qwen2.5-1.5B-Instruct")
out = pipe(messages, max_new_tokens=128)
print(out[0]["generated_text"][-1]["content"])

That is the recommended path, and it is why the message-list form is worth adopting even in throwaway scripts: it stops you hand-writing a format you will get wrong later.

Multi-turn, and the thing to get right

A conversation is the whole list, re-templated every turn:

messages.append({"role": "assistant", "content": reply})
messages.append({"role": "user", "content": follow_up})

Append the assistant’s clean reply, not the raw decode with special tokens still in it. Feeding <|im_end|> back in as message content corrupts the next template render, and the model starts emitting stray markers.

Tool calling lives here too

Models trained to call functions define the format in the same template. apply_chat_template(messages, tools=[...]) renders the tool schemas into whatever shape that family expects, and the model’s tool calls come back as text you parse. There is no cross-model standard, which is why a tool-calling application written against one model needs work to move to another. Frameworks paper over this; the underlying variation is real.

Two models, one application

If your code has to support more than one model, keep the messages list as the only representation of a conversation anywhere in your program, and render it at the last possible moment with whichever tokenizer is loaded. The moment a rendered string with [INST] in it is stored in a database or passed between functions, you have hard-coded one family, and swapping the model becomes a refactor rather than a config change.

The same rule protects your evaluation set. Store the messages, not the prompt. A set of rendered prompts measures one model and cannot be reused against its successor, which is the quiet way a team ends up unable to compare anything.

When there is no template

Base models have none, and apply_chat_template raises or falls back to a default. That error is useful information: it usually means you picked a base checkpoint when you wanted an instruct one. Occasionally a genuine instruct model ships without its template by oversight; then the card, or a discussion thread, is where the format is written, and it is worth opening a pull request to add it.

Do this now

Print the rendered template for two instruct models from different families with the same two messages. Put the outputs side by side. They will share no markers at all; and that is the whole argument for never writing the prompt format by hand.