What the model literally receives

The message list is not the prompt

You send a list of dictionaries. The model receives a single sequence of integers. Between those two facts sits a chat template, and it is where a surprising share of your token budget and several of your bugs live.

Here is the transformation, made visible with an open-weights model so you can run it yourself:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
msgs = [
    {"role": "system", "content": "You are terse."},
    {"role": "user", "content": "Hello"},
]
print(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))

The output is not You are terse. Hello. It is a marked-up string along the lines of:

<|im_start|>system
You are terse.<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant

Seven words of yours became roughly twenty tokens. The wrapper is small here and stops being small once you have twelve turns and six tools.

What the wrapper actually contains

Four things you did not write are in every request:

1. Turn delimiters. <|im_start|>, <|eot_id|>, [INST], depending on the family. Two to four tokens per message, every message, forever.
2. Role labels. The literal strings system, user, assistant, tool.
3. A generation prompt. The trailing tokens that tell the model it is now the assistant’s turn. Omit it in a local setup and the model will happily continue writing the user’s message instead: a classic first-day bug when people build their own serving loop.
4. Tool definitions, serialised. This is the expensive one. Your six JSON schemas are rendered into the prompt as text, usually with an instruction paragraph about how to call them. Hosted APIs do it server-side, which is why the cost is invisible until you read the usage numbers.

On a hosted API you never see this string. You still pay for it, and your token estimate is wrong by however much of it you ignored.

Roles are a convention, not a permission system

The most consequential fact in this lesson: the system role is not privileged at the level of the machine. It is a string in a particular position, wrapped in particular delimiters, that the model was trained to weight heavily.

That training is real and it works most of the time. It is also the entire basis of the system role’s authority. There is no flag in the tensor that says obey this one. Which is why a sufficiently insistent instruction inside a retrieved document sometimes wins, and why the defence against that is never “put it in the system prompt more firmly”.

The practical corollary: if you serialise a user’s text into your system prompt through string interpolation, you have handed them the strongest position in the request. Interpolating untrusted text into the system prompt is the single highest-severity assembly bug, and it is usually written by somebody trying to be helpful with an f-string.

Tool definitions cost real money

A modest tool schema (name, description, four parameters with descriptions) serialises to roughly 150 to 250 tokens. Six tools is around 1,200 tokens on every single call, whether or not any tool is used.

At a typical hosted input price and a million calls a month, that block alone is a line item worth arguing about. It is also usually the least edited text in the whole system, because it lives in code next to the function rather than in a prompt file that somebody owns.

Measure it directly:

# Send the same request with and without tools; the difference is the schema cost.
a = client.messages.count_tokens(model=M, messages=msgs)
b = client.messages.count_tokens(model=M, messages=msgs, tools=TOOLS)
print(b.input_tokens - a.input_tokens)

Multimodal blocks are tokens too

An image is not free and it is not one token. Most vision models cut an image into patches and each patch becomes a token or a small group of them. A 1024×1024 image commonly lands somewhere between 700 and 1,600 tokens depending on the model and how it tiles.

Two consequences. A four-page scanned PDF sent as images can cost more than the same document sent as extracted text, often several times more. And a chat where the user has pasted six screenshots is a context problem, not a UX problem.

Audio behaves similarly, priced per second of input rather than per patch, and video is currently the most expensive thing you can put in a window by a wide margin.

Prefill caching happens on this string

One more reason to know what the wrapper looks like: caching operates on the token sequence, including the delimiters. Two requests that differ only in a whitespace character inside the system content produce different token sequences and therefore two different cache entries.

This is why a template that stamps the current timestamp into the system prompt destroys the cache for the entire prefix below it. The lesson on caching does the arithmetic; the point here is that the arithmetic operates on this string, not on your message list.

The exercise

Take a real request from your system. Render it with apply_chat_template for the nearest open model you can find, print it, and read it as text. Almost everybody who does this the first time finds one of these: a stray blank line multiplying across turns, a tool description written as a note to a colleague rather than to a model, or an entire block they forgot they were sending.