{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"tokenizers-in-practice","title":"Your text is integers, and the count is not the word count","url":"https://addaly.com/learn/hugging-face/tokenizers-in-practice","position":12,"of":75,"minutes":10,"answer":"A tokenizer maps text to integers from a fixed vocabulary, and the tokens-per-word ratio depends on whose language the vocabulary was built from.","keyIdea":"A tokenizer maps text to integers from a fixed vocabulary, and the tokens-per-word ratio depends on whose language the vocabulary was built from, so the same sentence can cost two to four times as much in an Indic script as in English on an English-centric model."},"body":{"format":"markdown","text":"## Look at the integers once\n\n```python\nfrom transformers import AutoTokenizer\n\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\nenc = tok(\"Tokenisation is unforgiving\")\nprint(enc[\"input_ids\"])\nprint(tok.convert_ids_to_tokens(enc[\"input_ids\"]))\n```\n\nYou get something like `['Token', 'isation', 'Ġis', 'Ġun', 'forg', 'iving']`: six tokens for three words. The `Ġ` is a visible stand-in for a leading space, because byte-level tokenizers treat the space as part of the token that follows it. That is why `\"cat\"` and `\" cat\"` are different tokens with different ids, and why a stray trailing space in a prompt can change the output.\n\nA tokenizer is a fixed vocabulary (typically 32,000 to 256,000 entries) plus rules for splitting anything into pieces from that vocabulary. Common words get one token. Rare words get chopped. Nothing is ever unrepresentable, because the fallback is bytes.\n\n## The ratio, and why it is not the same for everyone\n\nFor English prose with an English-centric vocabulary, the working figure is about **1.3 tokens per word**, or roughly four characters per token. That number is where the rule of thumb \"750 words ≈ 1,000 tokens\" comes from.\n\nIt does not transfer. The same text in Hindi, Bengali, Tamil, Telugu or Marathi, tokenized by a model whose vocabulary was built mostly from English text, commonly runs two to four times as many tokens for the same meaning, because Devanagari and other Indic scripts fall back to multi-byte pieces. Try it:\n\n> Figure: One sentence, four tokenizers. Bars (tokens for the same twelve-word sentence): English, English-centric vocabulary 15; Hindi, English-centric vocabulary 47; Hindi, multilingual vocabulary 19; English, multilingual vocabulary 16. The same meaning, measured four ways. On an English-centric vocabulary the Hindi sentence costs three times as many tokens, which is three times the price per request, three times the context window used, and three times as many steps to get right. A multilingual vocabulary closes most of the gap.\n\n```python\nfor s in [\"Where is my parcel?\", \"मेरा पार्सल कहाँ है?\"]:\n    print(len(tok(s)[\"input_ids\"]), s)\n```\n\nThree consequences, all practical:\n\n1. **Cost.** Per-token pricing means the same question costs more in Hindi than in English on the same model. This is a real, measurable inequity in how these systems are priced, and it is a property of the vocabulary rather than of the language.\n2. **Context.** A 4,096-token window holds much less Hindi than English.\n3. **Quality.** More tokens per word means the model has more steps to get right for the same content.\n\nModels with multilingual vocabularies (Qwen, Gemma, Aya, IndicBERT and the muril family) narrow the gap substantially. Checking the ratio on your own text is a ten-second test that should inform model choice and almost never does.\n\n## Special tokens appear without being asked for\n\n```python\ntok = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\nprint(tok(\"hello\")[\"input_ids\"])              # [101, 7592, 102]\nprint(tok.convert_ids_to_tokens([101, 102]))  # ['[CLS]', '[SEP]']\n```\n\nYou wrote one word and got three integers. Encoder models wrap input in markers; decoder models often prepend a beginning-of-sequence token. `add_special_tokens=False` turns this off, which you want when you are assembling a sequence yourself and do not want a second BOS in the middle of it, a common bug when people concatenate a chat template with a manual encode.\n\n## Truncation is silent\n\n```python\nenc = tok(long_text, truncation=True, max_length=512)\n```\n\nWithout `truncation=True`, a too-long input raises or produces a tensor the model rejects. With it, the text is cut and nothing tells you. A classifier handed a 3,000-word complaint reads the first 512 tokens, which, for a complaint, is the polite introduction rather than the accusation at the end.\n\nIf that matters, either chunk deliberately, or pick a model with a longer window, or use `truncation_side=\"left\"` so you keep the end instead of the beginning.\n\n## Offsets: getting back to the characters\n\n```python\nenc = tok(\"Ravi lives in Pune\", return_offsets_mapping=True)\n```\n\nEach token comes with the character span it came from. This is how token classification output becomes highlighted text in a UI, and it only works with the fast (Rust) tokenizers, one more reason `tokenizer.json` in a repository is good news.\n\n## Two rules that prevent most tokenizer bugs\n\n- **Never mix a tokenizer with a model from a different repository.** The integers mean different things in different vocabularies, and nothing checks.\n- **If you add tokens, resize the embeddings.** `tok.add_tokens([\"<sku>\"])` followed by `model.resize_token_embeddings(len(tok))`. Skip the second line and you get an index error at best, and reads from uninitialised memory at worst.\n\n## Do this now\n\nTake one paragraph of your own text (in whatever language you actually work in) and count its tokens under two models: one English-centric like `gpt2`, one multilingual like `Qwen/Qwen2.5-0.5B`. Divide by the word count. That ratio is a cost multiplier you will carry through the rest of the course."},"check":{"question":"A support tool costs three times as much per message for Hindi users as for English users on the same model and the same message lengths. What explains it?","options":[{"text":"Multilingual models run a translation step internally before generating, which doubles or triples the tokens processed per request.","correct":false,"tempting":"Imagines an internal pivot-translation stage that these models do not perform."},{"text":"Devanagari text falls back to many short vocabulary pieces, so the same meaning becomes far more tokens and billing is per token.","correct":true},{"text":"Non-Latin scripts are stored as UTF-16 internally, so every character consumes two positions in the context window.","correct":false,"tempting":"Confuses an encoding detail with vocabulary construction; tokenizers work over bytes and learned merges, not fixed-width characters."},{"text":"Hindi prompts are longer in characters on average, so the difference is in what users write rather than in how it is tokenized.","correct":false,"tempting":"Attributes to user behaviour a gap that persists when message lengths are held constant, as they are here."}]},"prev":{"slug":"the-auto-classes","title":"AutoModel gives you numbers, not answers","url":"https://addaly.com/learn/hugging-face/the-auto-classes"},"next":{"slug":"padding-and-attention-masks","title":"Padding on the wrong side ruins generation quietly","url":"https://addaly.com/learn/hugging-face/padding-and-attention-masks"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Your text is integers, and the count is not the word count', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/tokenizers-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Your text is integers, and the count is not the word count","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"A tokenizer maps text to integers from a fixed vocabulary, and the tokens-per-word ratio depends on whose language the vocabulary was built from, so the same sentence can cost two to four times as much in an Indic script as in English on an English-centric model.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens for the same twelve-word sentence","title":"One sentence, four tokenizers","series":[{"label":"English, English-centric vocabulary","value":15},{"label":"Hindi, English-centric vocabulary","value":47},{"label":"Hindi, multilingual vocabulary","value":19,"highlight":true},{"label":"English, multilingual vocabulary","value":16}],"caption":"The same meaning, measured four ways. On an English-centric vocabulary the Hindi sentence costs three times as many tokens, which is three times the price per request, three times the context window used, and three times as many steps to get right. A multilingual vocabulary closes most of the gap."},"alt":"Figure: One sentence, four tokenizers. Bars (tokens for the same twelve-word sentence): English, English-centric vocabulary 15; Hindi, English-centric vocabulary 47; Hindi, multilingual vocabulary 19; English, multilingual vocabulary 16. The same meaning, measured four ways. On an English-centric vocabulary the Hindi sentence costs three times as many tokens, which is three times the price per request, three times the context window used, and three times as many steps to get right. A multilingual vocabulary closes most of the gap."}},"blocks":[{"t":"h2","id":"look-at-the-integers-once","runs":[{"text":"Look at the integers once"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoTokenizer\n\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\nenc = tok(\"Tokenisation is unforgiving\")\nprint(enc[\"input_ids\"])\nprint(tok.convert_ids_to_tokens(enc[\"input_ids\"]))"},{"t":"p","runs":[{"text":"You get something like "},{"text":"['Token', 'isation', 'Ġis', 'Ġun', 'forg', 'iving']","code":true},{"text":": six tokens for three words. The "},{"text":"Ġ","code":true},{"text":" is a visible stand-in for a leading space, because byte-level tokenizers treat the space as part of the token that follows it. That is why "},{"text":"\"cat\"","code":true},{"text":" and "},{"text":"\" cat\"","code":true},{"text":" are different tokens with different ids, and why a stray trailing space in a prompt can change the output."}]},{"t":"p","runs":[{"text":"A tokenizer is a fixed vocabulary (typically 32,000 to 256,000 entries) plus rules for splitting anything into pieces from that vocabulary. Common words get one token. Rare words get chopped. Nothing is ever unrepresentable, because the fallback is bytes."}]},{"t":"h2","id":"the-ratio-and-why-it-is-not-the-same-for-everyone","runs":[{"text":"The ratio, and why it is not the same for everyone"}]},{"t":"p","runs":[{"text":"For English prose with an English-centric vocabulary, the working figure is about "},{"text":"1.3 tokens per word","b":true},{"text":", or roughly four characters per token. That number is where the rule of thumb “750 words ≈ 1,000 tokens” comes from."}]},{"t":"p","runs":[{"text":"It does not transfer. The same text in Hindi, Bengali, Tamil, Telugu or Marathi, tokenized by a model whose vocabulary was built mostly from English text, commonly runs two to four times as many tokens for the same meaning, because Devanagari and other Indic scripts fall back to multi-byte pieces. Try it:"}]},{"t":"code","lang":"python","label":"Python","text":"for s in [\"Where is my parcel?\", \"मेरा पार्सल कहाँ है?\"]:\n    print(len(tok(s)[\"input_ids\"]), s)"},{"t":"p","runs":[{"text":"Three consequences, all practical:"}]},{"t":"ol","start":1,"items":[[{"text":"Cost.","b":true},{"text":" Per-token pricing means the same question costs more in Hindi than in English on the same model. This is a real, measurable inequity in how these systems are priced, and it is a property of the vocabulary rather than of the language."}],[{"text":"Context.","b":true},{"text":" A 4,096-token window holds much less Hindi than English."}],[{"text":"Quality.","b":true},{"text":" More tokens per word means the model has more steps to get right for the same content."}]]},{"t":"p","runs":[{"text":"Models with multilingual vocabularies (Qwen, Gemma, Aya, IndicBERT and the muril family) narrow the gap substantially. Checking the ratio on your own text is a ten-second test that should inform model choice and almost never does."}]},{"t":"h2","id":"special-tokens-appear-without-being-asked-for","runs":[{"text":"Special tokens appear without being asked for"}]},{"t":"code","lang":"python","label":"Python","text":"tok = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\nprint(tok(\"hello\")[\"input_ids\"])              # [101, 7592, 102]\nprint(tok.convert_ids_to_tokens([101, 102]))  # ['[CLS]', '[SEP]']"},{"t":"p","runs":[{"text":"You wrote one word and got three integers. Encoder models wrap input in markers; decoder models often prepend a beginning-of-sequence token. "},{"text":"add_special_tokens=False","code":true},{"text":" turns this off, which you want when you are assembling a sequence yourself and do not want a second BOS in the middle of it, a common bug when people concatenate a chat template with a manual encode."}]},{"t":"h2","id":"truncation-is-silent","runs":[{"text":"Truncation is silent"}]},{"t":"code","lang":"python","label":"Python","text":"enc = tok(long_text, truncation=True, max_length=512)"},{"t":"p","runs":[{"text":"Without "},{"text":"truncation=True","code":true},{"text":", a too-long input raises or produces a tensor the model rejects. With it, the text is cut and nothing tells you. A classifier handed a 3,000-word complaint reads the first 512 tokens, which, for a complaint, is the polite introduction rather than the accusation at the end."}]},{"t":"p","runs":[{"text":"If that matters, either chunk deliberately, or pick a model with a longer window, or use "},{"text":"truncation_side=\"left\"","code":true},{"text":" so you keep the end instead of the beginning."}]},{"t":"h2","id":"offsets-getting-back-to-the-characters","runs":[{"text":"Offsets: getting back to the characters"}]},{"t":"code","lang":"python","label":"Python","text":"enc = tok(\"Ravi lives in Pune\", return_offsets_mapping=True)"},{"t":"p","runs":[{"text":"Each token comes with the character span it came from. This is how token classification output becomes highlighted text in a UI, and it only works with the fast (Rust) tokenizers, one more reason "},{"text":"tokenizer.json","code":true},{"text":" in a repository is good news."}]},{"t":"h2","id":"two-rules-that-prevent-most-tokenizer-bugs","runs":[{"text":"Two rules that prevent most tokenizer bugs"}]},{"t":"ul","items":[[{"text":"Never mix a tokenizer with a model from a different repository.","b":true},{"text":" The integers mean different things in different vocabularies, and nothing checks."}],[{"text":"If you add tokens, resize the embeddings.","b":true},{"text":" "},{"text":"tok.add_tokens([\"<sku>\"])","code":true},{"text":" followed by "},{"text":"model.resize_token_embeddings(len(tok))","code":true},{"text":". Skip the second line and you get an index error at best, and reads from uninitialised memory at worst."}]]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Take one paragraph of your own text (in whatever language you actually work in) and count its tokens under two models: one English-centric like "},{"text":"gpt2","code":true},{"text":", one multilingual like "},{"text":"Qwen/Qwen2.5-0.5B","code":true},{"text":". Divide by the word count. That ratio is a cost multiplier you will carry through the rest of the course."}]}],"check":{"question":[{"text":"A support tool costs three times as much per message for Hindi users as for English users on the same model and the same message lengths. What explains it?"}],"options":[{"text":[{"text":"Multilingual models run a translation step internally before generating, which doubles or triples the tokens processed per request."}],"correct":false,"tempting":[{"text":"Imagines an internal pivot-translation stage that these models do not perform."}]},{"text":[{"text":"Devanagari text falls back to many short vocabulary pieces, so the same meaning becomes far more tokens and billing is per token."}],"correct":true,"tempting":null},{"text":[{"text":"Non-Latin scripts are stored as UTF-16 internally, so every character consumes two positions in the context window."}],"correct":false,"tempting":[{"text":"Confuses an encoding detail with vocabulary construction; tokenizers work over bytes and learned merges, not fixed-width characters."}]},{"text":[{"text":"Hindi prompts are longer in characters on average, so the difference is in what users write rather than in how it is tokenized."}],"correct":false,"tempting":[{"text":"Attributes to user behaviour a gap that persists when message lengths are held constant, as they are here."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"the-auto-classes","title":"AutoModel gives you numbers, not answers","minutes":9},"next":{"slug":"padding-and-attention-masks","title":"Padding on the wrong side ruins generation quietly","minutes":9}}}