Translation, and the languages nobody measures

What BLEU is counting

BLEU counts how many n-grams of the output appear in a reference translation, with a penalty for being too short. Everything that goes wrong with it follows from that description.

• One reference is not the set of correct answers. A sentence has many valid translations; matching one of them is what gets rewarded.
• Morphology is punished. In Hindi, Tamil, Turkish or Finnish, a correct translation that inflects differently from the reference shares fewer whole-word n-grams. The translation is right and the score is lower.
• Short segments are unstable. On a six-word sentence, matching or missing one bigram swings the score enormously, which is why BLEU is only meaningful over a corpus of hundreds of segments, never per sentence.
• Tokenisation changes the number. The same output can score 28 or 34 depending on how words were split, which is why any BLEU without its configuration is uninterpretable.

Use sacrebleu, which fixes the tokenisation and prints a signature you can quote:

pip install sacrebleu
sacrebleu ref.txt -i hyp.txt -m bleu chrf

chrF in that same command scores character n-grams instead of word n-grams. It handles inflection and languages without spaces far better, and for Indian languages it should be your default over BLEU. Neural metrics such as COMET correlate better still with human judgement, at the cost of running a model and of being opaque, and they were trained mostly on high-resource language pairs, so their reliability drops exactly where you most need help.

Humans, with a typology rather than a rating

Asking a bilingual reviewer “score this out of five” produces threes and fours. Asking them to mark errors produces data. The industry convention is MQM: mark each error with a category and a severity.

• Accuracy: mistranslation, omission, addition, untranslated text.
• Fluency: grammar, spelling, punctuation, register.
• Terminology: wrong term for your domain, inconsistent term use.

Weight severities (minor 1, major 5, critical 10) sum per thousand words, and report the profile as well as the total. “Fluency is clean, terminology is wrong 8 times per thousand words” tells you to build a glossary. A single score of 3.6 tells you nothing.

Build a challenge set for what actually breaks

Corpus-level scores hide the failures that generate complaints. Make twenty items each for the phenomena you know matter, and report a pass rate per phenomenon:

• Named entities: people, brands and places should not be translated or transliterated inconsistently.
• Numbers, dates and units: Indian digit grouping, currency symbols, 12- and 24-hour times.
• Register and honorifics: the tu/tum/aap distinction, formal versus casual address, which is a product decision that must be stated before it can be scored.
• Code-switching and transliteration: Hinglish written in Latin script is most of the real traffic for many Indian products, and it is absent from every standard benchmark.
• Gender: where the source is ambiguous, the target language forces a choice, and the model will guess. Test it explicitly (“the doctor said she”, “the nurse said he”) because the guess follows the training distribution and lands on the stereotype.

Two things that look like evidence and are not

Round-trip translation. Translating back into the source and comparing is intuitive and unsound: two errors can cancel, and the return leg has its own quality, so you are measuring a composition of two systems and attributing the result to one.

A benchmark score for a language nobody on the team reads. If no one can read Odia, a COMET score for Odia is a number with nothing behind it. That is not an argument for hiding the language; it is an argument for saying so.

The honest report for a multilingual product

For most of the world’s languages there is no reliable automatic metric and no pool of reviewers to hire quickly. A truthful report says which languages were measured, by whom, and which were not measured at all, rather than printing a metric that is technically computable and practically empty.

If a language is unmeasured, the mitigations are procedural rather than statistical: a visible way for users to report a bad translation, a rate limit on automatic publication, and a named speaker who reviews a sample each month. That is weaker than a measurement and stronger than pretending.