{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"adding-your-own-tokens","title":"Adding tokens, and initialising them properly","url":"https://addaly.com/learn/fine-tuning/adding-your-own-tokens","position":51,"of":84,"minutes":8,"answer":"New vocabulary rows are initialised randomly by default, which places them outside the region existing embeddings occupy, so set them to the mean of existing embeddings or of the tokens they stand for.","keyIdea":"New vocabulary rows are initialised randomly by default, which places them outside the region existing embeddings occupy, so set them to the mean of existing embeddings or of the tokens they stand for, and remember they stay frozen under LoRA unless embed_tokens and lm_head are in modules_to_save."},"body":{"format":"markdown","text":"## When it is worth it\n\nAdding vocabulary is not a routine step. The three cases where it earns its cost:\n\n1. **A control token.** A `<|tool_call|>` or `<|thinking|>` marker you want the model to emit unambiguously, which no existing token can be confused with.\n2. **A pad token** the base model lacks, as the previous lesson described.\n3. **A script or domain the tokenizer handles badly.** If your language tokenises at four tokens per word, extending the vocabulary with common sequences from your corpus can halve that, but this only pays off alongside continued pretraining with a substantial amount of text, because the new embeddings have to be learned from scratch.\n\nFor anything else (a product name, a set of category labels, a handful of domain terms) leave the vocabulary alone. The model represents multi-token strings perfectly well, and every added token is an untrained row in two large matrices.\n\n## The mechanics\n\n```python\ntok.add_special_tokens({\"additional_special_tokens\": [\"<|tool_call|>\"]})\nmodel.resize_token_embeddings(len(tok), pad_to_multiple_of=64)\n```\n\nThe `pad_to_multiple_of=64` rounds the vocabulary up to a multiple of 64. This is a throughput detail, not a correctness one: matrix dimensions that are multiples of 64 map cleanly onto tensor core tiles, and a vocabulary of 128,257 can be meaningfully slower than one of 128,320. The extra rows are never selected and cost a few megabytes.\n\n## The initialisation that matters\n\nBy default, `resize_token_embeddings` fills new rows with values drawn from the model's initialiser: small random numbers, unrelated to anything the model knows.\n\nThat is worse than it sounds. The existing embedding vectors occupy a particular region of the space with a particular norm; a random vector drawn from a naive distribution sits somewhere else entirely. On the input side the model receives a signal it has no representation for. On the output side the new token's logit is computed from a random row, which early in training can make it wildly over- or under-probable and produce large gradients.\n\nThe fix is to initialise the new rows near the centre of the existing distribution:\n\n```python\nemb = model.get_input_embeddings().weight.data\nout = model.get_output_embeddings().weight.data\nn = len(new_tokens)\nemb[-n:] = emb[:-n].mean(dim=0, keepdim=True)\nout[-n:] = out[:-n].mean(dim=0, keepdim=True)\n```\n\nBetter still, where the new token has a meaning expressible in existing tokens, initialise it as the mean of *those* tokens' embeddings. A `<|invoice|>` token initialised as the average of the embeddings for \"invoice\" starts in approximately the right place and converges much faster.\n\n## They will not train unless you say so\n\nWith a LoRA, the embedding table and output head are frozen like everything else. Your new token's embedding stays at whatever you initialised it to, forever.\n\n```python\nLoraConfig(..., modules_to_save=[\"embed_tokens\", \"lm_head\"])\n```\n\nThis trains those two matrices fully and saves them in the adapter. Be aware of the size: an 8B model's embedding table is 128,256 × 4,096 ≈ 525 million parameters, so your adapter file grows from about 160 MB to over 2 GB once both matrices are included. That is usually acceptable and it is always surprising the first time.\n\n## Tied embeddings\n\nMany models, particularly smaller ones, tie the input embedding matrix and the output head, using the same weights for both. Check `model.config.tie_word_embeddings`.\n\nWhen they are tied, you have one matrix, not two: resizing handles both together, `modules_to_save` needs only one entry, and the mean-initialisation above should be applied once. When they are untied, they are independent and both need attention. Assuming the wrong one is a quiet source of a new token that works on input and misbehaves on output.\n\n## After adding, before training\n\nThree checks:\n\n```python\nassert tok.convert_tokens_to_ids(\"<|tool_call|>\") != tok.unk_token_id\nassert len(tok) <= model.get_input_embeddings().weight.shape[0]\nprint(tok.tokenize(\"a <|tool_call|> b\"))   # must be one token, not five\n```\n\nThe third is the one that catches the real mistake. A special token added to the vocabulary but not registered as *special* will be split into its constituent characters by the tokenizer, and you will have trained the model to emit a seven-token sequence that only looks like a control token."},"check":{"question":"A team adds a <|tool_call|> token, trains a LoRA with the standard seven target modules, and finds the model never emits the new token. What did they miss?","options":[{"text":"The embedding table and output head are frozen under LoRA, so the new token's randomly initialised rows were never trained.","correct":true},{"text":"The token needs a corresponding entry in the chat template, without which the tokenizer strips it during templating.","correct":false,"tempting":"Names a template concern that would affect encoding of training data, not the model's inability to produce an output it was never trained on."},{"text":"New tokens are appended after the original vocabulary size, so the output head's softmax never assigns them probability mass.","correct":false,"tempting":"Invents a structural exclusion; after resizing, the head produces a logit for every row including the new ones."},{"text":"Special tokens are excluded from the loss by default, so the model received no gradient on positions where it should appear.","correct":false,"tempting":"Confuses masking of prompt tokens with a blanket exclusion of special tokens, which no standard collator performs."}]},"prev":{"slug":"the-tokenizer-is-the-model","title":"The tokenizer is part of the model","url":"https://addaly.com/learn/fine-tuning/the-tokenizer-is-the-model"},"next":{"slug":"a-config-line-by-line","title":"A working config, line by line","url":"https://addaly.com/learn/fine-tuning/a-config-line-by-line"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Adding tokens, and initialising them properly', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/adding-your-own-tokens (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Adding tokens, and initialising them properly","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"New vocabulary rows are initialised randomly by default, which places them outside the region existing embeddings occupy, so set them to the mean of existing embeddings or of the tokens they stand for, and remember they stay frozen under LoRA unless embed_tokens and lm_head are in modules_to_save.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"when-it-is-worth-it","runs":[{"text":"When it is worth it"}]},{"t":"p","runs":[{"text":"Adding vocabulary is not a routine step. The three cases where it earns its cost:"}]},{"t":"ol","start":1,"items":[[{"text":"A control token.","b":true},{"text":" A "},{"text":"<|tool_call|>","code":true},{"text":" or "},{"text":"<|thinking|>","code":true},{"text":" marker you want the model to emit unambiguously, which no existing token can be confused with."}],[{"text":"A pad token","b":true},{"text":" the base model lacks, as the previous lesson described."}],[{"text":"A script or domain the tokenizer handles badly.","b":true},{"text":" If your language tokenises at four tokens per word, extending the vocabulary with common sequences from your corpus can halve that, but this only pays off alongside continued pretraining with a substantial amount of text, because the new embeddings have to be learned from scratch."}]]},{"t":"p","runs":[{"text":"For anything else (a product name, a set of category labels, a handful of domain terms) leave the vocabulary alone. The model represents multi-token strings perfectly well, and every added token is an untrained row in two large matrices."}]},{"t":"h2","id":"the-mechanics","runs":[{"text":"The mechanics"}]},{"t":"code","lang":"python","label":"Python","text":"tok.add_special_tokens({\"additional_special_tokens\": [\"<|tool_call|>\"]})\nmodel.resize_token_embeddings(len(tok), pad_to_multiple_of=64)"},{"t":"p","runs":[{"text":"The "},{"text":"pad_to_multiple_of=64","code":true},{"text":" rounds the vocabulary up to a multiple of 64. This is a throughput detail, not a correctness one: matrix dimensions that are multiples of 64 map cleanly onto tensor core tiles, and a vocabulary of 128,257 can be meaningfully slower than one of 128,320. The extra rows are never selected and cost a few megabytes."}]},{"t":"h2","id":"the-initialisation-that-matters","runs":[{"text":"The initialisation that matters"}]},{"t":"p","runs":[{"text":"By default, "},{"text":"resize_token_embeddings","code":true},{"text":" fills new rows with values drawn from the model’s initialiser: small random numbers, unrelated to anything the model knows."}]},{"t":"p","runs":[{"text":"That is worse than it sounds. The existing embedding vectors occupy a particular region of the space with a particular norm; a random vector drawn from a naive distribution sits somewhere else entirely. On the input side the model receives a signal it has no representation for. On the output side the new token’s logit is computed from a random row, which early in training can make it wildly over- or under-probable and produce large gradients."}]},{"t":"p","runs":[{"text":"The fix is to initialise the new rows near the centre of the existing distribution:"}]},{"t":"code","lang":"python","label":"Python","text":"emb = model.get_input_embeddings().weight.data\nout = model.get_output_embeddings().weight.data\nn = len(new_tokens)\nemb[-n:] = emb[:-n].mean(dim=0, keepdim=True)\nout[-n:] = out[:-n].mean(dim=0, keepdim=True)"},{"t":"p","runs":[{"text":"Better still, where the new token has a meaning expressible in existing tokens, initialise it as the mean of "},{"text":"those","i":true},{"text":" tokens’ embeddings. A "},{"text":"<|invoice|>","code":true},{"text":" token initialised as the average of the embeddings for “invoice” starts in approximately the right place and converges much faster."}]},{"t":"h2","id":"they-will-not-train-unless-you-say-so","runs":[{"text":"They will not train unless you say so"}]},{"t":"p","runs":[{"text":"With a LoRA, the embedding table and output head are frozen like everything else. Your new token’s embedding stays at whatever you initialised it to, forever."}]},{"t":"code","lang":"python","label":"Python","text":"LoraConfig(..., modules_to_save=[\"embed_tokens\", \"lm_head\"])"},{"t":"p","runs":[{"text":"This trains those two matrices fully and saves them in the adapter. Be aware of the size: an 8B model’s embedding table is 128,256 × 4,096 ≈ 525 million parameters, so your adapter file grows from about 160 MB to over 2 GB once both matrices are included. That is usually acceptable and it is always surprising the first time."}]},{"t":"h2","id":"tied-embeddings","runs":[{"text":"Tied embeddings"}]},{"t":"p","runs":[{"text":"Many models, particularly smaller ones, tie the input embedding matrix and the output head, using the same weights for both. Check "},{"text":"model.config.tie_word_embeddings","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"When they are tied, you have one matrix, not two: resizing handles both together, "},{"text":"modules_to_save","code":true},{"text":" needs only one entry, and the mean-initialisation above should be applied once. When they are untied, they are independent and both need attention. Assuming the wrong one is a quiet source of a new token that works on input and misbehaves on output."}]},{"t":"h2","id":"after-adding-before-training","runs":[{"text":"After adding, before training"}]},{"t":"p","runs":[{"text":"Three checks:"}]},{"t":"code","lang":"python","label":"Python","text":"assert tok.convert_tokens_to_ids(\"<|tool_call|>\") != tok.unk_token_id\nassert len(tok) <= model.get_input_embeddings().weight.shape[0]\nprint(tok.tokenize(\"a <|tool_call|> b\"))   # must be one token, not five"},{"t":"p","runs":[{"text":"The third is the one that catches the real mistake. A special token added to the vocabulary but not registered as "},{"text":"special","i":true},{"text":" will be split into its constituent characters by the tokenizer, and you will have trained the model to emit a seven-token sequence that only looks like a control token."}]}],"check":{"question":[{"text":"A team adds a <|tool_call|> token, trains a LoRA with the standard seven target modules, and finds the model never emits the new token. What did they miss?"}],"options":[{"text":[{"text":"The embedding table and output head are frozen under LoRA, so the new token’s randomly initialised rows were never trained."}],"correct":true,"tempting":null},{"text":[{"text":"The token needs a corresponding entry in the chat template, without which the tokenizer strips it during templating."}],"correct":false,"tempting":[{"text":"Names a template concern that would affect encoding of training data, not the model’s inability to produce an output it was never trained on."}]},{"text":[{"text":"New tokens are appended after the original vocabulary size, so the output head’s softmax never assigns them probability mass."}],"correct":false,"tempting":[{"text":"Invents a structural exclusion; after resizing, the head produces a logit for every row including the new ones."}]},{"text":[{"text":"Special tokens are excluded from the loss by default, so the model received no gradient on positions where it should appear."}],"correct":false,"tempting":[{"text":"Confuses masking of prompt tokens with a blanket exclusion of special tokens, which no standard collator performs."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"the-tokenizer-is-the-model","title":"The tokenizer is part of the model","minutes":9},"next":{"slug":"a-config-line-by-line","title":"A working config, line by line","minutes":10}}}