Testing the language you actually work in

The largest undiscussed difference

Two models can sit within a point of each other on English benchmarks and be a world apart in Marathi, Yoruba, Vietnamese, Amharic or Quechua. This gap is bigger than almost any other difference between tools, it decides whether the tool is usable for a great many people, and it appears on no comparison page anywhere, because the people writing those pages work in English.

If you work in another language, this lesson is the most important test in the course for you.

Why the gap exists

Training mixture. Models are trained on text scraped largely from the web, and the web is wildly unequal by language. English is dominant, a dozen languages have substantial presence, and hundreds have very little. A model’s fluency broadly tracks how much of that language it saw.

Deliberate emphasis. Some makers invest specifically. Models from Chinese labs are usually stronger in Chinese; some are also unusually strong across Asian languages generally. European labs put weight on European languages. This is a real, checkable difference between families, not marketing.

The tokeniser, which also costs you money. Text is split into tokens, and tokenisers are fitted mostly to Latin-script text. The same sentence in Hindi in Devanagari script can take two to four times as many tokens as its English translation. That means three things at once: you pay several times more for the same content, you fit far less into the context window, and generation is slower. Try it in any tokeniser tool. The number surprises everybody the first time.

Six tests worth running

Ordinary drafting. A real message you would send. Judge whether it reads as written by a speaker or translated from English. Translated-English is the characteristic failure: grammatically correct, oddly formal, wrong idiom.

Register. Most languages carry formality distinctions English handles loosely: the pronoun choice, honorifics, the difference between how you address a customer and a cousin. Ask for the same message twice, formal and casual, and see whether the difference is real.

Code-switching. Hinglish, Spanglish, Taglish, Arabic-French mixing. A great deal of real communication does this, and models vary from natural to comically wrong. Give it a mixed input and see whether the reply mixes the same way.

Script and transliteration. Roman-script Hindi, Arabizi, romanised Cantonese. Many people type this way. Some models handle it fluently; some silently answer in the wrong script.

Reasoning in the language. Ask it to work out a problem in your language, not to translate a solution. Models often reason better in English and produce a weaker answer when made to think in another language. Worth knowing which of your tools does this.

Names, places, dates and numbers. Local name spellings, address formats, the lakh-crore system, date orders, currency conventions. A tool that renders 1,50,00,000 as 150,000 will cause you a real problem eventually.

The workaround that is not a failure

If the best tool for your task is weak in your language, a two-step approach is often better than either alone: work in English and translate the final output yourself, or use a dedicated translation system for the last step. It is not elegant and it is frequently the right answer, especially for technical or legal material where the reasoning matters more than the phrasing.

Free paths for the translation step are strong. NLLB-200 from Meta and MADLAD-400 from Google are openly released translation models covering 200 and 400 languages respectively, and both run on ordinary hardware. Argos Translate packages offline translation for the desktop. For everyday needs, the free tiers of the major translation services remain hard to beat.

One more thing to check: what it does with your language by default

Some products detect the language of your message and reply in it; some reply in English unless told otherwise; some reply in the language of their own system prompt. In a mixed-language conversation this gets erratic, and it is worth knowing which behaviour you are buying, because correcting the language every few messages is a genuine daily irritation.

Also check what happens on a long input. A twenty-page document in a non-Latin script consumes far more of the window than its English equivalent, so a tool that handles a fifty-page English contract comfortably may truncate the same contract in Bangla. Test with a real document rather than a paragraph.

Writing it down

Add a line to your prompt file recording which tool won in your language and on what date, because this is the fastest-moving part of the whole field. Open models from labs in India, China, Africa and the Gulf are being released specifically to close these gaps, and a language that was badly served eighteen months ago may be well served now by a model you can download for nothing.

Check it yourself. Nobody is going to check it for you.