The tokenizer is part of the model

Fixed, and consequential

The tokenizer is not a preprocessing convenience. It is a fixed component of the model, decided at pretraining time, and its vocabulary maps one-to-one onto the rows of the embedding table. Use a different tokenizer with the same weights and you are feeding the model random words.

So: save the tokenizer beside every checkpoint, load it from the same directory as the weights, and never assume two models in the same family share one.

What tokenisation costs different languages

The cost of a request is measured in tokens, and tokens are not distributed fairly.

A modern 128,000-entry vocabulary tokenises English at roughly 1.3 tokens per word. The same tokenizer on Devanagari, Bengali or Tamil script produces substantially more: often two to four times as many tokens for text conveying the same meaning, because the training corpus contained far less of those scripts and the merge rules never learned their common sequences. Older 32,000-entry vocabularies were much worse still, frequently splitting Indic words into individual characters.

Three consequences follow, and they are not abstract:

• Cost. The same question costs three times as much to answer in Hindi as in English on a per-token price.
• Context. A 4,000-token window holds a third as much Hindi as English.
• Quality. A word split into six fragments is harder to learn a representation for than one split into two. Long-range dependencies span more positions.

When you are choosing a base model for a non-English task, tokeniser efficiency on your language is a first-class criterion, and it takes one minute to check:

for name in ["meta-llama/Llama-3.1-8B", "Qwen/Qwen2.5-7B", "sarvamai/sarvam-1"]:
    t = AutoTokenizer.from_pretrained(name)
    print(name, len(t(sample_hindi_paragraph)["input_ids"]))

Run it on a paragraph of your actual text. The differences between model families are often larger than any quality difference you would get from fine-tuning.

The pad token trap

Many base models ship with no pad token, because pretraining packed sequences and never needed one. Training code needs one, so the near-universal workaround is:

tok.pad_token = tok.eos_token        # very common, and a trap

Here is the trap. Your collator masks padding positions with -100 so they contribute no loss. If pad and end-of-sequence are the same id, a mask built by matching on the pad id will also mask the real end-of-sequence token at the end of each answer.

The model then never receives a gradient teaching it to stop. It trains happily, the loss looks fine, and at inference it generates until it hits the token limit: answering your question, then answering a question you did not ask, then starting a new conversation with itself.

Two correct fixes:

1. Add a distinct pad token. tok.add_special_tokens({"pad_token": "<|pad|>"}), then resize embeddings. Cleanest, and it costs one embedding row.
2. Build the mask from the attention mask, not from the token id. Positions are padding because the collator padded them, and the attention mask already records that. This is what well-written collators do.

Either way, verify it: the verification is in the debugging lesson and takes five seconds.

The double-BOS bug

apply_chat_template adds the beginning-of-sequence token as part of the template. If you then pass the resulting string through the tokenizer with default settings, the tokenizer adds another one.

text = tok.apply_chat_template(msgs, tokenize=False)
ids  = tok(text)["input_ids"]          # BOS is now there twice
print(tok.convert_ids_to_tokens(ids[:3]))

A model trained with two BOS tokens and served with one (or the reverse) has a systematic mismatch at the very first position, which is the position everything else conditions on. The effect ranges from mild degradation to nonsense depending on the model.

The fix is tok(text, add_special_tokens=False) when the template has already added them, and the check is to print the first three tokens of a real example before every run.

The other mismatches worth knowing

• Padding side. Left for generation and for decoder-based classification; right is the default for training collators. Getting it wrong for classification means pooling a pad token’s hidden state.
• Fast and slow tokenizers can differ on unusual whitespace and unicode. Use the fast one consistently in both training and serving rather than mixing.
• Chat template changes between releases. A model’s template lives in its tokenizer config and repository maintainers do update it. If you fine-tuned against one version and serve with a later one, the special tokens around each turn may differ. Pin the tokenizer revision alongside the model revision.

None of these is intellectually difficult. All of them produce a model that is subtly worse for reasons that look like a training problem, and all of them are found by printing one templated example and reading it.