What these models really do in your language

Three tiers, and where the cliff is

Model cards list languages. The list conflates three very different states.

Well supported. English, and to varying degrees Chinese, Spanish, French, German, Portuguese and Russian. Fluent, idiomatic, reliable instruction following, reasonable factual coverage.

Present but thinner. Hindi, Arabic, Bengali, Indonesian, Vietnamese, Turkish, Japanese, Korean and others. Fluent-sounding output with more factual error, weaker instruction adherence, occasional grammatical oddity, and a noticeable drop on anything requiring reasoning in the language rather than translation into it.

Nominal. Most of the world’s languages. The model produces something, and a native speaker will find it stilted, wrong, or written in the wrong register. The list on the model card is often derived from what appeared in training data at any volume, not from an evaluation.

The cliff between the second and third tier is steep, and it is where most of the disappointment lives.

Script and register, not just language

Two details specific to how these models are trained, and both matter in South Asia in particular.

Script changes everything. Hindi written in Devanagari and Hindi written in Roman letters are, to the tokenizer and often to the model, different problems. A model may be decent at one and poor at the other depending on what its corpus contained: web forums are heavily romanised, formal publishing is not. Test both scripts separately.

Code-switching is normal and is tested by nobody. Real Hinglish mixes English words into Hindi grammar, in Roman script, with English technical vocabulary. Models handle this unevenly and no public benchmark covers it. If your users write that way, your evaluation set must too, and it will not resemble anything you can download.

Register. A model may produce grammatically correct Hindi that reads as translated newspaper prose to somebody who would have written it conversationally. This is invisible to automated metrics and obvious to a native speaker in one sentence.

The translation sandwich

A common strategy: translate the input to English, do the work, translate back. It works, and it has three costs worth naming.

• Two extra model calls of latency and expense.
• Culture-specific content degrades. Names, honorifics, local references, legal and administrative terms with no English equivalent.
• Errors compound across three stages, and the failure appears in the final output where it is hardest to attribute.

It remains the right answer when the task is language-neutral (classification into English labels, extracting a date) and the wrong answer when the output goes to a user in their language.

A better version when you can afford it: do the work in the original language with a model that supports it, and use translation only to let you check the output.

Measuring, since benchmarks will not

No public benchmark will tell you whether a model is good enough for your users in Marathi. The procedure that will, and it takes about two hours:

1. Collect 30 real inputs in each language you care about, in the script your users actually use.
2. Write what a good response looks like, or a one-line rubric.
3. Run three candidate models at temperature 0.
4. Have a native speaker rate each output on three points: acceptable, awkward but usable, wrong.
5. Count tokens per input for each model while you are there.

The token count is the cost decision and the ratings are the quality decision, and neither is available anywhere else. Repeat it when you change model, quantisation or provider, because quantisation damages lower-resource languages more than English, as an earlier lesson showed.

What to reach for when the answer is not good enough

• A language-focused model. The families in the earlier lesson exist for this and often win decisively.
• Fine-tuning on a few hundred examples in your language and register. This fixes register and format better than it fixes knowledge, which is usually what is wrong.
• Retrieval in the user’s language. Much of what looks like a language weakness is a knowledge weakness, and giving the model the right paragraph in the right language fixes both.
• A larger model. Multilingual quality scales with size more steeply than English quality does, so this helps more here than elsewhere, at the cost the earlier lessons described.

The habit that matters most: never accept a language claim without a speaker of that language reading twenty outputs. Fluency and correctness diverge sharply outside the first tier, and fluency is what fools a non-speaker.