{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-tokenizer-is-the-model","title":"The tokenizer is part of the model","url":"https://addaly.com/learn/fine-tuning/the-tokenizer-is-the-model","position":50,"of":84,"minutes":9,"answer":"The tokenizer is a fixed part of the model whose efficiency on your language decides cost, context and quality; and setting the pad token equal to the end-of-sequence token, then masking by id.","keyIdea":"The tokenizer is a fixed part of the model whose efficiency on your language decides cost, context and quality; and setting the pad token equal to the end-of-sequence token, then masking by id, removes the gradient that teaches the model to stop."},"body":{"format":"markdown","text":"## Fixed, and consequential\n\nThe tokenizer is not a preprocessing convenience. It is a fixed component of the model, decided at pretraining time, and its vocabulary maps one-to-one onto the rows of the embedding table. Use a different tokenizer with the same weights and you are feeding the model random words.\n\nSo: save the tokenizer beside every checkpoint, load it from the same directory as the weights, and never assume two models in the same family share one.\n\n## What tokenisation costs different languages\n\nThe cost of a request is measured in tokens, and tokens are not distributed fairly.\n\nA modern 128,000-entry vocabulary tokenises English at roughly 1.3 tokens per word. The same tokenizer on Devanagari, Bengali or Tamil script produces substantially more: often two to four times as many tokens for text conveying the same meaning, because the training corpus contained far less of those scripts and the merge rules never learned their common sequences. Older 32,000-entry vocabularies were much worse still, frequently splitting Indic words into individual characters.\n\nThree consequences follow, and they are not abstract:\n\n- **Cost.** The same question costs three times as much to answer in Hindi as in English on a per-token price.\n- **Context.** A 4,000-token window holds a third as much Hindi as English.\n- **Quality.** A word split into six fragments is harder to learn a representation for than one split into two. Long-range dependencies span more positions.\n\nWhen you are choosing a base model for a non-English task, tokeniser efficiency on your language is a first-class criterion, and it takes one minute to check:\n\n```python\nfor name in [\"meta-llama/Llama-3.1-8B\", \"Qwen/Qwen2.5-7B\", \"sarvamai/sarvam-1\"]:\n    t = AutoTokenizer.from_pretrained(name)\n    print(name, len(t(sample_hindi_paragraph)[\"input_ids\"]))\n```\n\nRun it on a paragraph of your actual text. The differences between model families are often larger than any quality difference you would get from fine-tuning.\n\n## The pad token trap\n\nMany base models ship with no pad token, because pretraining packed sequences and never needed one. Training code needs one, so the near-universal workaround is:\n\n```python\ntok.pad_token = tok.eos_token        # very common, and a trap\n```\n\nHere is the trap. Your collator masks padding positions with `-100` so they contribute no loss. If pad and end-of-sequence are the same id, a mask built by matching on the pad id will also mask **the real end-of-sequence token at the end of each answer**.\n\nThe model then never receives a gradient teaching it to stop. It trains happily, the loss looks fine, and at inference it generates until it hits the token limit: answering your question, then answering a question you did not ask, then starting a new conversation with itself.\n\nTwo correct fixes:\n\n1. **Add a distinct pad token.** `tok.add_special_tokens({\"pad_token\": \"<|pad|>\"})`, then resize embeddings. Cleanest, and it costs one embedding row.\n2. **Build the mask from the attention mask, not from the token id.** Positions are padding because the collator padded them, and the attention mask already records that. This is what well-written collators do.\n\nEither way, verify it: the verification is in the debugging lesson and takes five seconds.\n\n## The double-BOS bug\n\n`apply_chat_template` adds the beginning-of-sequence token as part of the template. If you then pass the resulting string through the tokenizer with default settings, the tokenizer adds another one.\n\n```python\ntext = tok.apply_chat_template(msgs, tokenize=False)\nids  = tok(text)[\"input_ids\"]          # BOS is now there twice\nprint(tok.convert_ids_to_tokens(ids[:3]))\n```\n\nA model trained with two BOS tokens and served with one (or the reverse) has a systematic mismatch at the very first position, which is the position everything else conditions on. The effect ranges from mild degradation to nonsense depending on the model.\n\nThe fix is `tok(text, add_special_tokens=False)` when the template has already added them, and the check is to print the first three tokens of a real example before every run.\n\n## The other mismatches worth knowing\n\n- **Padding side.** Left for generation and for decoder-based classification; right is the default for training collators. Getting it wrong for classification means pooling a pad token's hidden state.\n- **Fast and slow tokenizers** can differ on unusual whitespace and unicode. Use the fast one consistently in both training and serving rather than mixing.\n- **Chat template changes between releases.** A model's template lives in its tokenizer config and repository maintainers do update it. If you fine-tuned against one version and serve with a later one, the special tokens around each turn may differ. Pin the tokenizer revision alongside the model revision.\n\nNone of these is intellectually difficult. All of them produce a model that is subtly worse for reasons that look like a training problem, and all of them are found by printing one templated example and reading it."},"check":{"question":"A fine-tuned model answers correctly and then keeps generating, inventing a new user question and answering that too. The team set tok.pad_token = tok.eos_token and masks padding by matching the pad id. What is the cause?","options":[{"text":"The maximum generation length is set too high at inference, allowing the model to continue past a natural stopping point.","correct":false,"tempting":"Treats a missing stop signal as a sampling limit; a model that learned to emit an end token stops regardless of the ceiling."},{"text":"Packing merged examples without a separator, so the model learned to continue from one answer into the next prompt.","correct":false,"tempting":"Names a real cause of the same symptom, but the described setup points at masking rather than at packing."},{"text":"The eos token's embedding was reinitialised when it was reassigned as the pad token, destroying its learned representation.","correct":false,"tempting":"Assumes reassigning a role rewrites an embedding; setting pad_token changes a configuration field, not the embedding table."},{"text":"The mask matches every occurrence of the id, including the real end token ending each answer.","correct":true}]},"prev":{"slug":"the-run","title":"Running a LoRA, and What It Costs","url":"https://addaly.com/learn/fine-tuning/the-run"},"next":{"slug":"adding-your-own-tokens","title":"Adding tokens, and initialising them properly","url":"https://addaly.com/learn/fine-tuning/adding-your-own-tokens"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The tokenizer is part of the model', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-tokenizer-is-the-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The tokenizer is part of the model","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"The tokenizer is a fixed part of the model whose efficiency on your language decides cost, context and quality; and setting the pad token equal to the end-of-sequence token, then masking by id, removes the gradient that teaches the model to stop.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"fixed-and-consequential","runs":[{"text":"Fixed, and consequential"}]},{"t":"p","runs":[{"text":"The tokenizer is not a preprocessing convenience. It is a fixed component of the model, decided at pretraining time, and its vocabulary maps one-to-one onto the rows of the embedding table. Use a different tokenizer with the same weights and you are feeding the model random words."}]},{"t":"p","runs":[{"text":"So: save the tokenizer beside every checkpoint, load it from the same directory as the weights, and never assume two models in the same family share one."}]},{"t":"h2","id":"what-tokenisation-costs-different-languages","runs":[{"text":"What tokenisation costs different languages"}]},{"t":"p","runs":[{"text":"The cost of a request is measured in tokens, and tokens are not distributed fairly."}]},{"t":"p","runs":[{"text":"A modern 128,000-entry vocabulary tokenises English at roughly 1.3 tokens per word. The same tokenizer on Devanagari, Bengali or Tamil script produces substantially more: often two to four times as many tokens for text conveying the same meaning, because the training corpus contained far less of those scripts and the merge rules never learned their common sequences. Older 32,000-entry vocabularies were much worse still, frequently splitting Indic words into individual characters."}]},{"t":"p","runs":[{"text":"Three consequences follow, and they are not abstract:"}]},{"t":"ul","items":[[{"text":"Cost.","b":true},{"text":" The same question costs three times as much to answer in Hindi as in English on a per-token price."}],[{"text":"Context.","b":true},{"text":" A 4,000-token window holds a third as much Hindi as English."}],[{"text":"Quality.","b":true},{"text":" A word split into six fragments is harder to learn a representation for than one split into two. Long-range dependencies span more positions."}]]},{"t":"p","runs":[{"text":"When you are choosing a base model for a non-English task, tokeniser efficiency on your language is a first-class criterion, and it takes one minute to check:"}]},{"t":"code","lang":"python","label":"Python","text":"for name in [\"meta-llama/Llama-3.1-8B\", \"Qwen/Qwen2.5-7B\", \"sarvamai/sarvam-1\"]:\n    t = AutoTokenizer.from_pretrained(name)\n    print(name, len(t(sample_hindi_paragraph)[\"input_ids\"]))"},{"t":"p","runs":[{"text":"Run it on a paragraph of your actual text. The differences between model families are often larger than any quality difference you would get from fine-tuning."}]},{"t":"h2","id":"the-pad-token-trap","runs":[{"text":"The pad token trap"}]},{"t":"p","runs":[{"text":"Many base models ship with no pad token, because pretraining packed sequences and never needed one. Training code needs one, so the near-universal workaround is:"}]},{"t":"code","lang":"python","label":"Python","text":"tok.pad_token = tok.eos_token        # very common, and a trap"},{"t":"p","runs":[{"text":"Here is the trap. Your collator masks padding positions with "},{"text":"-100","code":true},{"text":" so they contribute no loss. If pad and end-of-sequence are the same id, a mask built by matching on the pad id will also mask "},{"text":"the real end-of-sequence token at the end of each answer","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"The model then never receives a gradient teaching it to stop. It trains happily, the loss looks fine, and at inference it generates until it hits the token limit: answering your question, then answering a question you did not ask, then starting a new conversation with itself."}]},{"t":"p","runs":[{"text":"Two correct fixes:"}]},{"t":"ol","start":1,"items":[[{"text":"Add a distinct pad token.","b":true},{"text":" "},{"text":"tok.add_special_tokens({\"pad_token\": \"<|pad|>\"})","code":true},{"text":", then resize embeddings. Cleanest, and it costs one embedding row."}],[{"text":"Build the mask from the attention mask, not from the token id.","b":true},{"text":" Positions are padding because the collator padded them, and the attention mask already records that. This is what well-written collators do."}]]},{"t":"p","runs":[{"text":"Either way, verify it: the verification is in the debugging lesson and takes five seconds."}]},{"t":"h2","id":"the-double-bos-bug","runs":[{"text":"The double-BOS bug"}]},{"t":"p","runs":[{"text":"apply_chat_template","code":true},{"text":" adds the beginning-of-sequence token as part of the template. If you then pass the resulting string through the tokenizer with default settings, the tokenizer adds another one."}]},{"t":"code","lang":"python","label":"Python","text":"text = tok.apply_chat_template(msgs, tokenize=False)\nids  = tok(text)[\"input_ids\"]          # BOS is now there twice\nprint(tok.convert_ids_to_tokens(ids[:3]))"},{"t":"p","runs":[{"text":"A model trained with two BOS tokens and served with one (or the reverse) has a systematic mismatch at the very first position, which is the position everything else conditions on. The effect ranges from mild degradation to nonsense depending on the model."}]},{"t":"p","runs":[{"text":"The fix is "},{"text":"tok(text, add_special_tokens=False)","code":true},{"text":" when the template has already added them, and the check is to print the first three tokens of a real example before every run."}]},{"t":"h2","id":"the-other-mismatches-worth-knowing","runs":[{"text":"The other mismatches worth knowing"}]},{"t":"ul","items":[[{"text":"Padding side.","b":true},{"text":" Left for generation and for decoder-based classification; right is the default for training collators. Getting it wrong for classification means pooling a pad token’s hidden state."}],[{"text":"Fast and slow tokenizers","b":true},{"text":" can differ on unusual whitespace and unicode. Use the fast one consistently in both training and serving rather than mixing."}],[{"text":"Chat template changes between releases.","b":true},{"text":" A model’s template lives in its tokenizer config and repository maintainers do update it. If you fine-tuned against one version and serve with a later one, the special tokens around each turn may differ. Pin the tokenizer revision alongside the model revision."}]]},{"t":"p","runs":[{"text":"None of these is intellectually difficult. All of them produce a model that is subtly worse for reasons that look like a training problem, and all of them are found by printing one templated example and reading it."}]}],"check":{"question":[{"text":"A fine-tuned model answers correctly and then keeps generating, inventing a new user question and answering that too. The team set tok.pad_token = tok.eos_token and masks padding by matching the pad id. What is the cause?"}],"options":[{"text":[{"text":"The maximum generation length is set too high at inference, allowing the model to continue past a natural stopping point."}],"correct":false,"tempting":[{"text":"Treats a missing stop signal as a sampling limit; a model that learned to emit an end token stops regardless of the ceiling."}]},{"text":[{"text":"Packing merged examples without a separator, so the model learned to continue from one answer into the next prompt."}],"correct":false,"tempting":[{"text":"Names a real cause of the same symptom, but the described setup points at masking rather than at packing."}]},{"text":[{"text":"The eos token’s embedding was reinitialised when it was reassigned as the pad token, destroying its learned representation."}],"correct":false,"tempting":[{"text":"Assumes reassigning a role rewrites an embedding; setting pad_token changes a configuration field, not the embedding table."}]},{"text":[{"text":"The mask matches every occurrence of the id, including the real end token ending each answer."}],"correct":true,"tempting":null}],"order":[1,3,2,0],"why":null},"prev":{"slug":"the-run","title":"Running a LoRA, and What It Costs","minutes":10},"next":{"slug":"adding-your-own-tokens","title":"Adding tokens, and initialising them properly","minutes":8}}}