{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"tokenizers-across-families","title":"Why the same sentence costs different amounts","url":"https://addaly.com/learn/open-models/tokenizers-across-families","position":23,"of":71,"minutes":9,"answer":"Fertility (tokens per word) varies by a factor of three or more across languages and tokenizer vocabularies, and it multiplies straight into your bill, your effective context length and your latency.","keyIdea":"Fertility (tokens per word) varies by a factor of three or more across languages and tokenizer vocabularies, and it multiplies straight into your bill, your effective context length and your latency."},"body":{"format":"markdown","text":"## Tokens are the unit of everything\n\nPrice is per token. Context length is in tokens. Speed is tokens per second. So the tokenizer (a lookup table and a merge algorithm, no neural network involved) quietly sets the cost of your product, and it does so differently for every language.\n\nAlmost every current model uses byte-level byte-pair encoding or a SentencePiece variant. The training procedure is simple: start from bytes, repeatedly merge the most frequent adjacent pair, stop at the target vocabulary size. Whatever was frequent in the tokenizer's training text gets a short representation. Whatever was rare gets chopped into pieces.\n\nVocabulary sizes across the families: Llama 2 used 32,000; Llama 3 moved to 128,256; Qwen uses around 151,000; Gemma uses 256,000. That progression is mostly about multilingual coverage.\n\n## The number to measure: fertility\n\nFertility is tokens per word. Measure it on your own text, because it is the whole story:\n\n```python\nfrom transformers import AutoTokenizer\ntext_en = \"The delivery was delayed by three days and nobody informed me.\"\ntext_hi = \"डिलीवरी तीन दिन देर से हुई और किसी ने मुझे नहीं बताया।\"\n\nfor name in [\"meta-llama/Llama-2-7b-hf\",\n             \"meta-llama/Llama-3.1-8B\",\n             \"Qwen/Qwen3-8B\",\n             \"google/gemma-3-4b-it\"]:\n    t = AutoTokenizer.from_pretrained(name)\n    print(name, len(t.encode(text_en)), len(t.encode(text_hi)))\n```\n\n> Figure: Tokens per word for the same Hindi sentence in Devanagari. Bars (tokens per word): Llama 2: 32,000 vocabulary 6.2; Llama 3: 128,256 2.7; Qwen3: about 151,000 2.5; Gemma 3: 256,000 2.1. English costs about 1.3 tokens per word on all four. At three times the fertility you pay three times per sentence on an identical price list, the same window holds a third of the document, and a reply of the same length takes three times as long. Measure this on your own text before you choose.\n\nThe pattern you will see: English costs roughly 1.2 to 1.4 tokens per word everywhere. Hindi in Devanagari on an older 32k tokenizer can cost five to eight tokens per word, because each Devanagari character is three bytes in UTF-8 and, with few merges learned for those byte sequences, the tokenizer falls back to something close to byte-per-token. A newer large-vocabulary tokenizer typically brings that down to two or three.\n\nThree consequences follow directly, and none of them is about quality of writing:\n\n- **Cost.** At three times the fertility you pay three times as much per sentence, on identical hardware and an identical price list.\n- **Context.** A 32,000-token window holds perhaps 24,000 English words, or 6,000 Hindi words on a bad tokenizer. The same document does not fit.\n- **Speed.** Generation is per token, so a reply of the same length takes three times as long.\n\nThis is the mechanical part of why a language-tuned model is often the right choice: the tokenizer usually comes with it.\n\n## Numbers, code and the details that surprise\n\n**Digits.** Llama 3 splits numbers into groups of up to three digits; several other families split every digit separately. This is deliberate (consistent digit tokenisation makes arithmetic more learnable) and it means a long number is many tokens. It also means that arithmetic failures are partly a representation problem, not only a reasoning one.\n\n**Leading spaces.** In byte-level BPE, ` hello` and `hello` are different tokens. This is behind a whole family of odd bugs when you build prompts by string concatenation, and behind the trick called token healing that some libraries apply.\n\n**Code.** Indentation matters: tokenizers trained with code merge runs of spaces, so four spaces can be one token or four depending on the family. On a large codebase this is a real cost difference.\n\n**Whitespace and newlines** are tokens too, which is why heavily formatted prompts with many blank lines cost more than they look.\n\n## The tokenizer is part of the model\n\nYou cannot mix them. Loading Qwen weights with a Llama tokenizer produces text that is grammatical nonsense, because token 5000 means something different in each. When you download a model, take `tokenizer.json` and `tokenizer_config.json` from the same revision as the weights.\n\nThis also constrains fine-tuning. You can add tokens to a vocabulary, but every added token needs a new row in the embedding matrix and in the output head, initialised somehow, and the model has never seen those rows. Adding vocabulary for a new language is a real technique and it requires continued pretraining to be worth anything. A later lesson covers it.\n\n## A five-minute exercise worth doing once\n\nTake a thousand real documents from your own system. Run them through the tokenizers of the three models you are considering. Compute the mean tokens per document for each.\n\nThat number, multiplied by your request volume and the provider's price, is your bill. Teams routinely discover a 30% cost difference between two models of the same size that they were choosing between on quality alone."},"check":{"question":"A team serving Hindi content moves from a model with a 32,000-token vocabulary to one with 151,000, keeping the same parameter count and the same per-token price. Costs fall by about 60%. Why?","options":[{"text":"The larger vocabulary lets the model answer more concisely, so it generates fewer tokens per reply.","correct":false,"tempting":"Reply length is a behavioural property of the tuning; the saving here comes from how the same text is segmented, not from shorter answers."},{"text":"Larger vocabularies allow more aggressive batching, which reduces the per-request cost on the provider's side.","correct":false,"tempting":"Batching efficiency is set by memory and sequence length, and providers price per token regardless."},{"text":"The larger vocabulary learned merges for Devanagari byte sequences, so the same Hindi text becomes far fewer tokens.","correct":true},{"text":"The bigger tokenizer compresses the model's weights, so each token is cheaper to process.","correct":false,"tempting":"Confuses text segmentation with weight storage; vocabulary size increases the embedding matrices rather than compressing anything."}]},"prev":{"slug":"reading-config-json","title":"Every number in config.json","url":"https://addaly.com/learn/open-models/reading-config-json"},"next":{"slug":"the-chat-template-contract","title":"The chat template is a contract","url":"https://addaly.com/learn/open-models/the-chat-template-contract"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why the same sentence costs different amounts', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/tokenizers-across-families (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why the same sentence costs different amounts","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"Fertility (tokens per word) varies by a factor of three or more across languages and tokenizer vocabularies, and it multiplies straight into your bill, your effective context length and your latency.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens per word","title":"Tokens per word for the same Hindi sentence in Devanagari","series":[{"label":"Llama 2: 32,000 vocabulary","value":6.2,"highlight":true},{"label":"Llama 3: 128,256","value":2.7},{"label":"Qwen3: about 151,000","value":2.5},{"label":"Gemma 3: 256,000","value":2.1}],"caption":"English costs about 1.3 tokens per word on all four. At three times the fertility you pay three times per sentence on an identical price list, the same window holds a third of the document, and a reply of the same length takes three times as long. Measure this on your own text before you choose."},"alt":"Figure: Tokens per word for the same Hindi sentence in Devanagari. Bars (tokens per word): Llama 2: 32,000 vocabulary 6.2; Llama 3: 128,256 2.7; Qwen3: about 151,000 2.5; Gemma 3: 256,000 2.1. English costs about 1.3 tokens per word on all four. At three times the fertility you pay three times per sentence on an identical price list, the same window holds a third of the document, and a reply of the same length takes three times as long. Measure this on your own text before you choose."}},"blocks":[{"t":"h2","id":"tokens-are-the-unit-of-everything","runs":[{"text":"Tokens are the unit of everything"}]},{"t":"p","runs":[{"text":"Price is per token. Context length is in tokens. Speed is tokens per second. So the tokenizer (a lookup table and a merge algorithm, no neural network involved) quietly sets the cost of your product, and it does so differently for every language."}]},{"t":"p","runs":[{"text":"Almost every current model uses byte-level byte-pair encoding or a SentencePiece variant. The training procedure is simple: start from bytes, repeatedly merge the most frequent adjacent pair, stop at the target vocabulary size. Whatever was frequent in the tokenizer’s training text gets a short representation. Whatever was rare gets chopped into pieces."}]},{"t":"p","runs":[{"text":"Vocabulary sizes across the families: Llama 2 used 32,000; Llama 3 moved to 128,256; Qwen uses around 151,000; Gemma uses 256,000. That progression is mostly about multilingual coverage."}]},{"t":"h2","id":"the-number-to-measure-fertility","runs":[{"text":"The number to measure: fertility"}]},{"t":"p","runs":[{"text":"Fertility is tokens per word. Measure it on your own text, because it is the whole story:"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoTokenizer\ntext_en = \"The delivery was delayed by three days and nobody informed me.\"\ntext_hi = \"डिलीवरी तीन दिन देर से हुई और किसी ने मुझे नहीं बताया।\"\n\nfor name in [\"meta-llama/Llama-2-7b-hf\",\n             \"meta-llama/Llama-3.1-8B\",\n             \"Qwen/Qwen3-8B\",\n             \"google/gemma-3-4b-it\"]:\n    t = AutoTokenizer.from_pretrained(name)\n    print(name, len(t.encode(text_en)), len(t.encode(text_hi)))"},{"t":"p","runs":[{"text":"The pattern you will see: English costs roughly 1.2 to 1.4 tokens per word everywhere. Hindi in Devanagari on an older 32k tokenizer can cost five to eight tokens per word, because each Devanagari character is three bytes in UTF-8 and, with few merges learned for those byte sequences, the tokenizer falls back to something close to byte-per-token. A newer large-vocabulary tokenizer typically brings that down to two or three."}]},{"t":"p","runs":[{"text":"Three consequences follow directly, and none of them is about quality of writing:"}]},{"t":"ul","items":[[{"text":"Cost.","b":true},{"text":" At three times the fertility you pay three times as much per sentence, on identical hardware and an identical price list."}],[{"text":"Context.","b":true},{"text":" A 32,000-token window holds perhaps 24,000 English words, or 6,000 Hindi words on a bad tokenizer. The same document does not fit."}],[{"text":"Speed.","b":true},{"text":" Generation is per token, so a reply of the same length takes three times as long."}]]},{"t":"p","runs":[{"text":"This is the mechanical part of why a language-tuned model is often the right choice: the tokenizer usually comes with it."}]},{"t":"h2","id":"numbers-code-and-the-details-that-surprise","runs":[{"text":"Numbers, code and the details that surprise"}]},{"t":"p","runs":[{"text":"Digits.","b":true},{"text":" Llama 3 splits numbers into groups of up to three digits; several other families split every digit separately. This is deliberate (consistent digit tokenisation makes arithmetic more learnable) and it means a long number is many tokens. It also means that arithmetic failures are partly a representation problem, not only a reasoning one."}]},{"t":"p","runs":[{"text":"Leading spaces.","b":true},{"text":" In byte-level BPE, "},{"text":" hello","code":true},{"text":" and "},{"text":"hello","code":true},{"text":" are different tokens. This is behind a whole family of odd bugs when you build prompts by string concatenation, and behind the trick called token healing that some libraries apply."}]},{"t":"p","runs":[{"text":"Code.","b":true},{"text":" Indentation matters: tokenizers trained with code merge runs of spaces, so four spaces can be one token or four depending on the family. On a large codebase this is a real cost difference."}]},{"t":"p","runs":[{"text":"Whitespace and newlines","b":true},{"text":" are tokens too, which is why heavily formatted prompts with many blank lines cost more than they look."}]},{"t":"h2","id":"the-tokenizer-is-part-of-the-model","runs":[{"text":"The tokenizer is part of the model"}]},{"t":"p","runs":[{"text":"You cannot mix them. Loading Qwen weights with a Llama tokenizer produces text that is grammatical nonsense, because token 5000 means something different in each. When you download a model, take "},{"text":"tokenizer.json","code":true},{"text":" and "},{"text":"tokenizer_config.json","code":true},{"text":" from the same revision as the weights."}]},{"t":"p","runs":[{"text":"This also constrains fine-tuning. You can add tokens to a vocabulary, but every added token needs a new row in the embedding matrix and in the output head, initialised somehow, and the model has never seen those rows. Adding vocabulary for a new language is a real technique and it requires continued pretraining to be worth anything. A later lesson covers it."}]},{"t":"h2","id":"a-five-minute-exercise-worth-doing-once","runs":[{"text":"A five-minute exercise worth doing once"}]},{"t":"p","runs":[{"text":"Take a thousand real documents from your own system. Run them through the tokenizers of the three models you are considering. Compute the mean tokens per document for each."}]},{"t":"p","runs":[{"text":"That number, multiplied by your request volume and the provider’s price, is your bill. Teams routinely discover a 30% cost difference between two models of the same size that they were choosing between on quality alone."}]}],"check":{"question":[{"text":"A team serving Hindi content moves from a model with a 32,000-token vocabulary to one with 151,000, keeping the same parameter count and the same per-token price. Costs fall by about 60%. Why?"}],"options":[{"text":[{"text":"The larger vocabulary lets the model answer more concisely, so it generates fewer tokens per reply."}],"correct":false,"tempting":[{"text":"Reply length is a behavioural property of the tuning; the saving here comes from how the same text is segmented, not from shorter answers."}]},{"text":[{"text":"Larger vocabularies allow more aggressive batching, which reduces the per-request cost on the provider’s side."}],"correct":false,"tempting":[{"text":"Batching efficiency is set by memory and sequence length, and providers price per token regardless."}]},{"text":[{"text":"The larger vocabulary learned merges for Devanagari byte sequences, so the same Hindi text becomes far fewer tokens."}],"correct":true,"tempting":null},{"text":[{"text":"The bigger tokenizer compresses the model’s weights, so each token is cheaper to process."}],"correct":false,"tempting":[{"text":"Confuses text segmentation with weight storage; vocabulary size increases the embedding matrices rather than compressing anything."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"reading-config-json","title":"Every number in config.json","minutes":10},"next":{"slug":"the-chat-template-contract","title":"The chat template is a contract","minutes":9}}}