Why it is worse in Hindi, and costs more

Start with the bill

Take one paragraph, write it in English and in Hindi, and count the tokens each way.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
en = "The train to Lucknow leaves at six tomorrow morning."
hi = "लखनऊ की ट्रेन कल सुबह छह बजे निकलती है।"
print(len(tok.encode(en)), len(tok.encode(hi)))

On tokenizers from the 2020 to 2023 generation the Hindi version costs three to eight times as many tokens for the same content. On the newer, larger vocabularies (the 128,000-entry and 200,000-entry tokenizers that shipped with 2024 models) the ratio has come down to somewhere between one and a half and two and a half. Tamil, Bengali, Telugu and Kannada behave similarly, and some are worse.

Everything else in this lesson follows from that ratio, so it is worth being clear about why it exists.

The mechanism, from module one

Byte-pair encoding merges the most frequent adjacent pairs in its training corpus and stops at a fixed vocabulary size. English was most of that corpus, so English words became single tokens. Hindi was a fraction of a per cent of it, so Devanagari sequences earned few merges.

It is worse than “few merges” for scripts outside Latin. Byte-level tokenizers start from UTF-8 bytes, and a Devanagari character is three bytes. With no merges, one character is three tokens; a single common word can be a dozen. Every merge the tokenizer did learn for Hindi was a merge it could afford only because it had already covered English.

The consequences arrive in four places at once:

• Cost. The same conversation is billed at the token ratio. A support bot serving Hindi at two and a half times the English token count costs two and a half times as much per exchange.
• Context. A 128,000-token window holds a third to a half as much Hindi text. Module four’s budgeting has to be redone per language.
• Latency. Decode is one token per step, so responses take proportionally longer to stream.
• Length control. “Write 200 words” was already unreliable; at unfamiliar token boundaries it is worse.

Then the data

Module five: knowledge comes from pretraining, and the reversal-curse lesson showed that a fact needs many exposures to be reliably known. Web-derived corpora are roughly half English. Hindi, the language of several hundred million people, is well under one per cent of them, and most Indian languages are far smaller still.

So the model has seen a tiny fraction as much Hindi as English. Fewer exposures mean weaker facts, poorer idiom, less coverage of regional vocabulary, and (the previous lesson’s point) a safety training set that was mostly English too. Every published multilingual evaluation shows a gap between English and Indian languages on the same model, and the gap is widest on the tasks that need the most knowledge.

The latent-English finding

Point the logit lens from module two at a model answering in French, Chinese or Hindi. In one 2024 study of an open model, the intermediate layers decoded to English tokens (the English word for the concept being processed) before the final layers produced the target language. The interpretation is contested at the edges, and the plain reading is hard to avoid: the model’s internal computation leans on English, and other languages are handled partly as a translation layer around it.

If that is right, two practical things follow. Reasoning quality tracks the English path, which is why asking for the working in English and only the final answer in Hindi often measures better than reasoning in Hindi throughout. And errors compound: a concept that translates loosely gets reasoned about loosely. Models trained with a multilingual mix show less of this. Measure it on the model you are using rather than assuming either way.

Hinglish

Romanised Hindi (Hindi words in Latin letters, mixed with English) is a third distribution. It has no standard spelling, so the tokenizer sees English-looking fragments and the model has seen it mainly in chat and social text. Models are often surprisingly good at casual Hinglish conversation and poor at anything formal in it, and the safety gap from the previous lesson applies here too. If your users write this way, and in India a great many do, test in it specifically.

What to do

1. Measure fertility. Tokens per hundred characters, for your language and for English, on the tokenizer you will actually use. That single number sets your cost, context and latency budgets.
2. Evaluate in the language you serve. An English benchmark score is not a Hindi score. Build the evaluation set from Hindi inputs, including Hinglish if that is what arrives.
3. Try reason-in-English, answer-in-Hindi, and measure whether it helps on your task. It often does; it is not free of translation errors.
4. Choose the tokenizer when you self-host. Between two models of similar quality, the one with the larger multilingual vocabulary can be two to three times cheaper to run for Indian languages, and module seven’s arithmetic makes that concrete.
5. Use the Indic-first tools. Translation models built for Indian languages, such as AI4Bharat’s IndicTrans2, are free, run on a CPU, and often beat a general model at translation specifically. Indian-language chat models from Indian labs exist and are worth including in any comparison. The free path is a local translation model plus a local general model, and it is a real option for a laptop.

The honest state

The gap is closing: each generation’s tokenizer covers more scripts and each training mix includes more non-English text, and the ratio that was five has become two. It is not closed. On every published measurement, quality in Hindi lags English on the same model, and the lag is largest where knowledge matters most. Anyone building for India should budget tokens for the language they serve and measure quality in it, rather than reading the English numbers and dividing by hope.