{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-translation","title":"Translation, and the languages nobody measures","url":"https://addaly.com/learn/evaluating-ai/evaluating-translation","position":31,"of":76,"minutes":10,"answer":"BLEU rewards matching one reference's word forms, so it under-rates morphologically rich languages and short segments: use chrF alongside it, test named phenomena separately, and say plainly when a language is simply unmeasured.","keyIdea":"BLEU rewards matching one reference's word forms, so it under-rates morphologically rich languages and short segments: use chrF alongside it, test named phenomena separately, and say plainly when a language is simply unmeasured."},"body":{"format":"markdown","text":"## What BLEU is counting\n\nBLEU counts how many n-grams of the output appear in a reference translation, with a penalty for being too short. Everything that goes wrong with it follows from that description.\n\n- **One reference is not the set of correct answers.** A sentence has many valid translations; matching one of them is what gets rewarded.\n- **Morphology is punished.** In Hindi, Tamil, Turkish or Finnish, a correct translation that inflects differently from the reference shares fewer whole-word n-grams. The translation is right and the score is lower.\n- **Short segments are unstable.** On a six-word sentence, matching or missing one bigram swings the score enormously, which is why BLEU is only meaningful over a corpus of hundreds of segments, never per sentence.\n- **Tokenisation changes the number.** The same output can score 28 or 34 depending on how words were split, which is why any BLEU without its configuration is uninterpretable.\n\nUse `sacrebleu`, which fixes the tokenisation and prints a signature you can quote:\n\n```bash\npip install sacrebleu\nsacrebleu ref.txt -i hyp.txt -m bleu chrf\n```\n\n**chrF** in that same command scores character n-grams instead of word n-grams. It handles inflection and languages without spaces far better, and for Indian languages it should be your default over BLEU. Neural metrics such as COMET correlate better still with human judgement, at the cost of running a model and of being opaque, and they were trained mostly on high-resource language pairs, so their reliability drops exactly where you most need help.\n\n## Humans, with a typology rather than a rating\n\nAsking a bilingual reviewer \"score this out of five\" produces threes and fours. Asking them to mark errors produces data. The industry convention is MQM: mark each error with a category and a severity.\n\n- **Accuracy**: mistranslation, omission, addition, untranslated text.\n- **Fluency**: grammar, spelling, punctuation, register.\n- **Terminology**: wrong term for your domain, inconsistent term use.\n\nWeight severities (minor 1, major 5, critical 10) sum per thousand words, and report the profile as well as the total. \"Fluency is clean, terminology is wrong 8 times per thousand words\" tells you to build a glossary. A single score of 3.6 tells you nothing.\n\n## Build a challenge set for what actually breaks\n\nCorpus-level scores hide the failures that generate complaints. Make twenty items each for the phenomena you know matter, and report a pass rate per phenomenon:\n\n- **Named entities**: people, brands and places should not be translated or transliterated inconsistently.\n- **Numbers, dates and units**: Indian digit grouping, currency symbols, 12- and 24-hour times.\n- **Register and honorifics**: the tu/tum/aap distinction, formal versus casual address, which is a product decision that must be stated before it can be scored.\n- **Code-switching and transliteration**: Hinglish written in Latin script is most of the real traffic for many Indian products, and it is absent from every standard benchmark.\n- **Gender**: where the source is ambiguous, the target language forces a choice, and the model will guess. Test it explicitly (\"the doctor said she\", \"the nurse said he\") because the guess follows the training distribution and lands on the stereotype.\n\n## Two things that look like evidence and are not\n\n**Round-trip translation.** Translating back into the source and comparing is intuitive and unsound: two errors can cancel, and the return leg has its own quality, so you are measuring a composition of two systems and attributing the result to one.\n\n**A benchmark score for a language nobody on the team reads.** If no one can read Odia, a COMET score for Odia is a number with nothing behind it. That is not an argument for hiding the language; it is an argument for saying so.\n\n## The honest report for a multilingual product\n\nFor most of the world's languages there is no reliable automatic metric and no pool of reviewers to hire quickly. A truthful report says which languages were measured, by whom, and which were not measured at all, rather than printing a metric that is technically computable and practically empty.\n\nIf a language is unmeasured, the mitigations are procedural rather than statistical: a visible way for users to report a bad translation, a rate limit on automatic publication, and a named speaker who reviews a sample each month. That is weaker than a measurement and stronger than pretending."},"check":{"question":"A team reports BLEU 31 for English-to-Hindi and BLEU 42 for English-to-German, and concludes the system is much weaker in Hindi. What is the main flaw in that conclusion?","options":[{"text":"BLEU scores are not comparable across language pairs, partly because word-level n-gram matching penalises Hindi's richer inflection, so the gap may reflect morphology rather than quality.","correct":true},{"text":"BLEU requires at least four reference translations per segment, and the team appears to have used one.","correct":false,"tempting":"Multiple references do help, but single-reference BLEU is standard practice and does not explain a cross-language gap."},{"text":"Hindi text needs to be transliterated to Latin script before BLEU can be computed correctly.","correct":false,"tempting":"Invents a requirement; sacreBLEU handles Devanagari directly, and transliterating would introduce error rather than remove it."},{"text":"The German test set was probably larger, and BLEU rises with corpus size.","correct":false,"tempting":"BLEU is a corpus-level ratio, not a quantity that grows with the number of segments, so size alone does not shift it in a systematic direction."}]},"prev":{"slug":"evaluating-summaries","title":"Summaries: what was invented, and what was left out","url":"https://addaly.com/learn/evaluating-ai/evaluating-summaries"},"next":{"slug":"evaluating-code-generation","title":"Code: run it, in a box, against tests it has not seen","url":"https://addaly.com/learn/evaluating-ai/evaluating-code-generation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Translation, and the languages nobody measures', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-translation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Translation, and the languages nobody measures","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"BLEU rewards matching one reference’s word forms, so it under-rates morphologically rich languages and short segments: use chrF alongside it, test named phenomena separately, and say plainly when a language is simply unmeasured.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-bleu-is-counting","runs":[{"text":"What BLEU is counting"}]},{"t":"p","runs":[{"text":"BLEU counts how many n-grams of the output appear in a reference translation, with a penalty for being too short. Everything that goes wrong with it follows from that description."}]},{"t":"ul","items":[[{"text":"One reference is not the set of correct answers.","b":true},{"text":" A sentence has many valid translations; matching one of them is what gets rewarded."}],[{"text":"Morphology is punished.","b":true},{"text":" In Hindi, Tamil, Turkish or Finnish, a correct translation that inflects differently from the reference shares fewer whole-word n-grams. The translation is right and the score is lower."}],[{"text":"Short segments are unstable.","b":true},{"text":" On a six-word sentence, matching or missing one bigram swings the score enormously, which is why BLEU is only meaningful over a corpus of hundreds of segments, never per sentence."}],[{"text":"Tokenisation changes the number.","b":true},{"text":" The same output can score 28 or 34 depending on how words were split, which is why any BLEU without its configuration is uninterpretable."}]]},{"t":"p","runs":[{"text":"Use "},{"text":"sacrebleu","code":true},{"text":", which fixes the tokenisation and prints a signature you can quote:"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install sacrebleu\nsacrebleu ref.txt -i hyp.txt -m bleu chrf"},{"t":"p","runs":[{"text":"chrF","b":true},{"text":" in that same command scores character n-grams instead of word n-grams. It handles inflection and languages without spaces far better, and for Indian languages it should be your default over BLEU. Neural metrics such as COMET correlate better still with human judgement, at the cost of running a model and of being opaque, and they were trained mostly on high-resource language pairs, so their reliability drops exactly where you most need help."}]},{"t":"h2","id":"humans-with-a-typology-rather-than-a-rating","runs":[{"text":"Humans, with a typology rather than a rating"}]},{"t":"p","runs":[{"text":"Asking a bilingual reviewer “score this out of five” produces threes and fours. Asking them to mark errors produces data. The industry convention is MQM: mark each error with a category and a severity."}]},{"t":"ul","items":[[{"text":"Accuracy","b":true},{"text":": mistranslation, omission, addition, untranslated text."}],[{"text":"Fluency","b":true},{"text":": grammar, spelling, punctuation, register."}],[{"text":"Terminology","b":true},{"text":": wrong term for your domain, inconsistent term use."}]]},{"t":"p","runs":[{"text":"Weight severities (minor 1, major 5, critical 10) sum per thousand words, and report the profile as well as the total. “Fluency is clean, terminology is wrong 8 times per thousand words” tells you to build a glossary. A single score of 3.6 tells you nothing."}]},{"t":"h2","id":"build-a-challenge-set-for-what-actually-breaks","runs":[{"text":"Build a challenge set for what actually breaks"}]},{"t":"p","runs":[{"text":"Corpus-level scores hide the failures that generate complaints. Make twenty items each for the phenomena you know matter, and report a pass rate per phenomenon:"}]},{"t":"ul","items":[[{"text":"Named entities","b":true},{"text":": people, brands and places should not be translated or transliterated inconsistently."}],[{"text":"Numbers, dates and units","b":true},{"text":": Indian digit grouping, currency symbols, 12- and 24-hour times."}],[{"text":"Register and honorifics","b":true},{"text":": the tu/tum/aap distinction, formal versus casual address, which is a product decision that must be stated before it can be scored."}],[{"text":"Code-switching and transliteration","b":true},{"text":": Hinglish written in Latin script is most of the real traffic for many Indian products, and it is absent from every standard benchmark."}],[{"text":"Gender","b":true},{"text":": where the source is ambiguous, the target language forces a choice, and the model will guess. Test it explicitly (“the doctor said she”, “the nurse said he”) because the guess follows the training distribution and lands on the stereotype."}]]},{"t":"h2","id":"two-things-that-look-like-evidence-and-are-not","runs":[{"text":"Two things that look like evidence and are not"}]},{"t":"p","runs":[{"text":"Round-trip translation.","b":true},{"text":" Translating back into the source and comparing is intuitive and unsound: two errors can cancel, and the return leg has its own quality, so you are measuring a composition of two systems and attributing the result to one."}]},{"t":"p","runs":[{"text":"A benchmark score for a language nobody on the team reads.","b":true},{"text":" If no one can read Odia, a COMET score for Odia is a number with nothing behind it. That is not an argument for hiding the language; it is an argument for saying so."}]},{"t":"h2","id":"the-honest-report-for-a-multilingual-product","runs":[{"text":"The honest report for a multilingual product"}]},{"t":"p","runs":[{"text":"For most of the world’s languages there is no reliable automatic metric and no pool of reviewers to hire quickly. A truthful report says which languages were measured, by whom, and which were not measured at all, rather than printing a metric that is technically computable and practically empty."}]},{"t":"p","runs":[{"text":"If a language is unmeasured, the mitigations are procedural rather than statistical: a visible way for users to report a bad translation, a rate limit on automatic publication, and a named speaker who reviews a sample each month. That is weaker than a measurement and stronger than pretending."}]}],"check":{"question":[{"text":"A team reports BLEU 31 for English-to-Hindi and BLEU 42 for English-to-German, and concludes the system is much weaker in Hindi. What is the main flaw in that conclusion?"}],"options":[{"text":[{"text":"BLEU scores are not comparable across language pairs, partly because word-level n-gram matching penalises Hindi’s richer inflection, so the gap may reflect morphology rather than quality."}],"correct":true,"tempting":null},{"text":[{"text":"BLEU requires at least four reference translations per segment, and the team appears to have used one."}],"correct":false,"tempting":[{"text":"Multiple references do help, but single-reference BLEU is standard practice and does not explain a cross-language gap."}]},{"text":[{"text":"Hindi text needs to be transliterated to Latin script before BLEU can be computed correctly."}],"correct":false,"tempting":[{"text":"Invents a requirement; sacreBLEU handles Devanagari directly, and transliterating would introduce error rather than remove it."}]},{"text":[{"text":"The German test set was probably larger, and BLEU rises with corpus size."}],"correct":false,"tempting":[{"text":"BLEU is a corpus-level ratio, not a quantity that grows with the number of segments, so size alone does not shift it in a systematic direction."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"evaluating-summaries","title":"Summaries: what was invented, and what was left out","minutes":9},"next":{"slug":"evaluating-code-generation","title":"Code: run it, in a box, against tests it has not seen","minutes":9}}}