{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"language-imbalance","title":"Why it is worse in Hindi, and costs more","url":"https://addaly.com/learn/how-llms-work/language-imbalance","position":58,"of":78,"minutes":9,"answer":"A tokenizer trained mostly on English splits Devanagari into several times as many tokens, which multiplies cost, halves the usable context and slows every response.","keyIdea":"A tokenizer trained mostly on English splits Devanagari into several times as many tokens, which multiplies cost, halves the usable context and slows every response, and the same data imbalance that caused it also thins the model's knowledge and safety training in that language."},"body":{"format":"markdown","text":"## Start with the bill\n\nTake one paragraph, write it in English and in Hindi, and count the tokens each way.\n\n```python\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-hf\")\nen = \"The train to Lucknow leaves at six tomorrow morning.\"\nhi = \"लखनऊ की ट्रेन कल सुबह छह बजे निकलती है।\"\nprint(len(tok.encode(en)), len(tok.encode(hi)))\n```\n\nOn tokenizers from the 2020 to 2023 generation the Hindi version costs three to eight times as many tokens for the same content. On the newer, larger vocabularies (the 128,000-entry and 200,000-entry tokenizers that shipped with 2024 models) the ratio has come down to somewhere between one and a half and two and a half. Tamil, Bengali, Telugu and Kannada behave similarly, and some are worse.\n\nEverything else in this lesson follows from that ratio, so it is worth being clear about why it exists.\n\n## The mechanism, from module one\n\nByte-pair encoding merges the most frequent adjacent pairs in its training corpus and stops at a fixed vocabulary size. English was most of that corpus, so English words became single tokens. Hindi was a fraction of a per cent of it, so Devanagari sequences earned few merges.\n\nIt is worse than \"few merges\" for scripts outside Latin. Byte-level tokenizers start from UTF-8 bytes, and a Devanagari character is three bytes. With no merges, one character is three tokens; a single common word can be a dozen. Every merge the tokenizer did learn for Hindi was a merge it could afford only because it had already covered English.\n\nThe consequences arrive in four places at once:\n\n- **Cost.** The same conversation is billed at the token ratio. A support bot serving Hindi at two and a half times the English token count costs two and a half times as much per exchange.\n- **Context.** A 128,000-token window holds a third to a half as much Hindi text. Module four's budgeting has to be redone per language.\n- **Latency.** Decode is one token per step, so responses take proportionally longer to stream.\n- **Length control.** \"Write 200 words\" was already unreliable; at unfamiliar token boundaries it is worse.\n\n## Then the data\n\nModule five: knowledge comes from pretraining, and the reversal-curse lesson showed that a fact needs many exposures to be reliably known. Web-derived corpora are roughly half English. Hindi, the language of several hundred million people, is well under one per cent of them, and most Indian languages are far smaller still.\n\nSo the model has seen a tiny fraction as much Hindi as English. Fewer exposures mean weaker facts, poorer idiom, less coverage of regional vocabulary, and (the previous lesson's point) a safety training set that was mostly English too. Every published multilingual evaluation shows a gap between English and Indian languages on the same model, and the gap is widest on the tasks that need the most knowledge.\n\n> Figure: Share of a web-derived pretraining corpus. Bars (% of documents): English 49; Other European languages 27; Chinese 5.2; Japanese 2.4; Hindi 0.6; All other Indian languages together 0.5. Several hundred million Hindi speakers, and well under one per cent of the documents. Fewer exposures mean weaker facts, thinner idiom and a safety training set that was mostly English too, which is why an English benchmark score is not a Hindi score and has to be measured again in the language you actually serve.\n\n## The latent-English finding\n\nPoint the logit lens from module two at a model answering in French, Chinese or Hindi. In one 2024 study of an open model, the intermediate layers decoded to English tokens (the English word for the concept being processed) before the final layers produced the target language. The interpretation is contested at the edges, and the plain reading is hard to avoid: the model's internal computation leans on English, and other languages are handled partly as a translation layer around it.\n\nIf that is right, two practical things follow. Reasoning quality tracks the English path, which is why asking for the working in English and only the final answer in Hindi often measures better than reasoning in Hindi throughout. And errors compound: a concept that translates loosely gets reasoned about loosely. Models trained with a multilingual mix show less of this. Measure it on the model you are using rather than assuming either way.\n\n## Hinglish\n\nRomanised Hindi (Hindi words in Latin letters, mixed with English) is a third distribution. It has no standard spelling, so the tokenizer sees English-looking fragments and the model has seen it mainly in chat and social text. Models are often surprisingly good at casual Hinglish conversation and poor at anything formal in it, and the safety gap from the previous lesson applies here too. If your users write this way, and in India a great many do, test in it specifically.\n\n## What to do\n\n1. **Measure fertility.** Tokens per hundred characters, for your language and for English, on the tokenizer you will actually use. That single number sets your cost, context and latency budgets.\n2. **Evaluate in the language you serve.** An English benchmark score is not a Hindi score. Build the evaluation set from Hindi inputs, including Hinglish if that is what arrives.\n3. **Try reason-in-English, answer-in-Hindi**, and measure whether it helps on your task. It often does; it is not free of translation errors.\n4. **Choose the tokenizer when you self-host.** Between two models of similar quality, the one with the larger multilingual vocabulary can be two to three times cheaper to run for Indian languages, and module seven's arithmetic makes that concrete.\n5. **Use the Indic-first tools.** Translation models built for Indian languages, such as AI4Bharat's IndicTrans2, are free, run on a CPU, and often beat a general model at translation specifically. Indian-language chat models from Indian labs exist and are worth including in any comparison. The free path is a local translation model plus a local general model, and it is a real option for a laptop.\n\n## The honest state\n\nThe gap is closing: each generation's tokenizer covers more scripts and each training mix includes more non-English text, and the ratio that was five has become two. It is not closed. On every published measurement, quality in Hindi lags English on the same model, and the lag is largest where knowledge matters most. Anyone building for India should budget tokens for the language they serve and measure quality in it, rather than reading the English numbers and dividing by hope."},"check":{"question":"A team's Hindi chatbot costs 2.4 times as much per conversation as its English twin and hits the context limit much sooner, with identical prompts and conversation lengths. What is the primary mechanism?","options":[{"text":"The model is more verbose in Hindi, producing longer replies for the same question.","correct":false,"tempting":"Verbosity varies by language a little, but the ratio appears on the input side too, on text the team wrote themselves, which verbosity cannot explain."},{"text":"The tokenizer's merges were learned mostly on English, so Devanagari text splits into far more tokens per unit of content.","correct":true},{"text":"Providers bill non-ASCII characters at a premium because they take more bytes to transmit and store.","correct":false,"tempting":"Bytes are the right neighbourhood (a Devanagari character is three UTF-8 bytes) but billing is per token, and the byte count matters only through how the tokenizer turns bytes into tokens."},{"text":"The model translates the conversation into English internally, reasons there, and translates back, and both hidden translations are counted against the token bill alongside the visible text.","correct":false,"tempting":"Misapplies the latent-English finding, which is about intermediate representations inside a forward pass; nothing is billed for what happens between layers."}]},"prev":{"slug":"why-jailbreaks-work","title":"Why jailbreaks work: two failures of safety training","url":"https://addaly.com/learn/how-llms-work/why-jailbreaks-work"},"next":{"slug":"the-bandwidth-wall","title":"The bandwidth wall: why decode speed is a division","url":"https://addaly.com/learn/how-llms-work/the-bandwidth-wall"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why it is worse in Hindi, and costs more', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/language-imbalance (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why it is worse in Hindi, and costs more","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"A tokenizer trained mostly on English splits Devanagari into several times as many tokens, which multiplies cost, halves the usable context and slows every response, and the same data imbalance that caused it also thins the model’s knowledge and safety training in that language.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of documents","title":"Share of a web-derived pretraining corpus","series":[{"label":"English","value":49},{"label":"Other European languages","value":27},{"label":"Chinese","value":5.2},{"label":"Japanese","value":2.4},{"label":"Hindi","value":0.6,"highlight":true},{"label":"All other Indian languages together","value":0.5}],"caption":"Several hundred million Hindi speakers, and well under one per cent of the documents. Fewer exposures mean weaker facts, thinner idiom and a safety training set that was mostly English too, which is why an English benchmark score is not a Hindi score and has to be measured again in the language you actually serve."},"alt":"Figure: Share of a web-derived pretraining corpus. Bars (% of documents): English 49; Other European languages 27; Chinese 5.2; Japanese 2.4; Hindi 0.6; All other Indian languages together 0.5. Several hundred million Hindi speakers, and well under one per cent of the documents. Fewer exposures mean weaker facts, thinner idiom and a safety training set that was mostly English too, which is why an English benchmark score is not a Hindi score and has to be measured again in the language you actually serve."}},"blocks":[{"t":"h2","id":"start-with-the-bill","runs":[{"text":"Start with the bill"}]},{"t":"p","runs":[{"text":"Take one paragraph, write it in English and in Hindi, and count the tokens each way."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-hf\")\nen = \"The train to Lucknow leaves at six tomorrow morning.\"\nhi = \"लखनऊ की ट्रेन कल सुबह छह बजे निकलती है।\"\nprint(len(tok.encode(en)), len(tok.encode(hi)))"},{"t":"p","runs":[{"text":"On tokenizers from the 2020 to 2023 generation the Hindi version costs three to eight times as many tokens for the same content. On the newer, larger vocabularies (the 128,000-entry and 200,000-entry tokenizers that shipped with 2024 models) the ratio has come down to somewhere between one and a half and two and a half. Tamil, Bengali, Telugu and Kannada behave similarly, and some are worse."}]},{"t":"p","runs":[{"text":"Everything else in this lesson follows from that ratio, so it is worth being clear about why it exists."}]},{"t":"h2","id":"the-mechanism-from-module-one","runs":[{"text":"The mechanism, from module one"}]},{"t":"p","runs":[{"text":"Byte-pair encoding merges the most frequent adjacent pairs in its training corpus and stops at a fixed vocabulary size. English was most of that corpus, so English words became single tokens. Hindi was a fraction of a per cent of it, so Devanagari sequences earned few merges."}]},{"t":"p","runs":[{"text":"It is worse than “few merges” for scripts outside Latin. Byte-level tokenizers start from UTF-8 bytes, and a Devanagari character is three bytes. With no merges, one character is three tokens; a single common word can be a dozen. Every merge the tokenizer did learn for Hindi was a merge it could afford only because it had already covered English."}]},{"t":"p","runs":[{"text":"The consequences arrive in four places at once:"}]},{"t":"ul","items":[[{"text":"Cost.","b":true},{"text":" The same conversation is billed at the token ratio. A support bot serving Hindi at two and a half times the English token count costs two and a half times as much per exchange."}],[{"text":"Context.","b":true},{"text":" A 128,000-token window holds a third to a half as much Hindi text. Module four’s budgeting has to be redone per language."}],[{"text":"Latency.","b":true},{"text":" Decode is one token per step, so responses take proportionally longer to stream."}],[{"text":"Length control.","b":true},{"text":" “Write 200 words” was already unreliable; at unfamiliar token boundaries it is worse."}]]},{"t":"h2","id":"then-the-data","runs":[{"text":"Then the data"}]},{"t":"p","runs":[{"text":"Module five: knowledge comes from pretraining, and the reversal-curse lesson showed that a fact needs many exposures to be reliably known. Web-derived corpora are roughly half English. Hindi, the language of several hundred million people, is well under one per cent of them, and most Indian languages are far smaller still."}]},{"t":"p","runs":[{"text":"So the model has seen a tiny fraction as much Hindi as English. Fewer exposures mean weaker facts, poorer idiom, less coverage of regional vocabulary, and (the previous lesson’s point) a safety training set that was mostly English too. Every published multilingual evaluation shows a gap between English and Indian languages on the same model, and the gap is widest on the tasks that need the most knowledge."}]},{"t":"h2","id":"the-latent-english-finding","runs":[{"text":"The latent-English finding"}]},{"t":"p","runs":[{"text":"Point the logit lens from module two at a model answering in French, Chinese or Hindi. In one 2024 study of an open model, the intermediate layers decoded to English tokens (the English word for the concept being processed) before the final layers produced the target language. The interpretation is contested at the edges, and the plain reading is hard to avoid: the model’s internal computation leans on English, and other languages are handled partly as a translation layer around it."}]},{"t":"p","runs":[{"text":"If that is right, two practical things follow. Reasoning quality tracks the English path, which is why asking for the working in English and only the final answer in Hindi often measures better than reasoning in Hindi throughout. And errors compound: a concept that translates loosely gets reasoned about loosely. Models trained with a multilingual mix show less of this. Measure it on the model you are using rather than assuming either way."}]},{"t":"h2","id":"hinglish","runs":[{"text":"Hinglish"}]},{"t":"p","runs":[{"text":"Romanised Hindi (Hindi words in Latin letters, mixed with English) is a third distribution. It has no standard spelling, so the tokenizer sees English-looking fragments and the model has seen it mainly in chat and social text. Models are often surprisingly good at casual Hinglish conversation and poor at anything formal in it, and the safety gap from the previous lesson applies here too. If your users write this way, and in India a great many do, test in it specifically."}]},{"t":"h2","id":"what-to-do","runs":[{"text":"What to do"}]},{"t":"ol","start":1,"items":[[{"text":"Measure fertility.","b":true},{"text":" Tokens per hundred characters, for your language and for English, on the tokenizer you will actually use. That single number sets your cost, context and latency budgets."}],[{"text":"Evaluate in the language you serve.","b":true},{"text":" An English benchmark score is not a Hindi score. Build the evaluation set from Hindi inputs, including Hinglish if that is what arrives."}],[{"text":"Try reason-in-English, answer-in-Hindi","b":true},{"text":", and measure whether it helps on your task. It often does; it is not free of translation errors."}],[{"text":"Choose the tokenizer when you self-host.","b":true},{"text":" Between two models of similar quality, the one with the larger multilingual vocabulary can be two to three times cheaper to run for Indian languages, and module seven’s arithmetic makes that concrete."}],[{"text":"Use the Indic-first tools.","b":true},{"text":" Translation models built for Indian languages, such as AI4Bharat’s IndicTrans2, are free, run on a CPU, and often beat a general model at translation specifically. Indian-language chat models from Indian labs exist and are worth including in any comparison. The free path is a local translation model plus a local general model, and it is a real option for a laptop."}]]},{"t":"h2","id":"the-honest-state","runs":[{"text":"The honest state"}]},{"t":"p","runs":[{"text":"The gap is closing: each generation’s tokenizer covers more scripts and each training mix includes more non-English text, and the ratio that was five has become two. It is not closed. On every published measurement, quality in Hindi lags English on the same model, and the lag is largest where knowledge matters most. Anyone building for India should budget tokens for the language they serve and measure quality in it, rather than reading the English numbers and dividing by hope."}]}],"check":{"question":[{"text":"A team’s Hindi chatbot costs 2.4 times as much per conversation as its English twin and hits the context limit much sooner, with identical prompts and conversation lengths. What is the primary mechanism?"}],"options":[{"text":[{"text":"The model is more verbose in Hindi, producing longer replies for the same question."}],"correct":false,"tempting":[{"text":"Verbosity varies by language a little, but the ratio appears on the input side too, on text the team wrote themselves, which verbosity cannot explain."}]},{"text":[{"text":"The tokenizer’s merges were learned mostly on English, so Devanagari text splits into far more tokens per unit of content."}],"correct":true,"tempting":null},{"text":[{"text":"Providers bill non-ASCII characters at a premium because they take more bytes to transmit and store."}],"correct":false,"tempting":[{"text":"Bytes are the right neighbourhood (a Devanagari character is three UTF-8 bytes) but billing is per token, and the byte count matters only through how the tokenizer turns bytes into tokens."}]},{"text":[{"text":"The model translates the conversation into English internally, reasons there, and translates back, and both hidden translations are counted against the token bill alongside the visible text."}],"correct":false,"tempting":[{"text":"Misapplies the latent-English finding, which is about intermediate representations inside a forward pass; nothing is billed for what happens between layers."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"why-jailbreaks-work","title":"Why jailbreaks work: two failures of safety training","minutes":9},"next":{"slug":"the-bandwidth-wall","title":"The bandwidth wall: why decode speed is a division","minutes":9}}}