{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"multilingual-reality","title":"What these models really do in your language","url":"https://addaly.com/learn/open-models/multilingual-reality","position":47,"of":71,"minutes":9,"answer":"Language support divides into well-supported, thin and nominal tiers with a steep cliff between the last two, script and register matter as much as language.","keyIdea":"Language support divides into well-supported, thin and nominal tiers with a steep cliff between the last two, script and register matter as much as language, and only a native speaker rating thirty real inputs will tell you which tier you are in."},"body":{"format":"markdown","text":"## Three tiers, and where the cliff is\n\nModel cards list languages. The list conflates three very different states.\n\n**Well supported.** English, and to varying degrees Chinese, Spanish, French, German, Portuguese and Russian. Fluent, idiomatic, reliable instruction following, reasonable factual coverage.\n\n**Present but thinner.** Hindi, Arabic, Bengali, Indonesian, Vietnamese, Turkish, Japanese, Korean and others. Fluent-sounding output with more factual error, weaker instruction adherence, occasional grammatical oddity, and a noticeable drop on anything requiring reasoning in the language rather than translation into it.\n\n**Nominal.** Most of the world's languages. The model produces something, and a native speaker will find it stilted, wrong, or written in the wrong register. The list on the model card is often derived from what appeared in training data at any volume, not from an evaluation.\n\nThe cliff between the second and third tier is steep, and it is where most of the disappointment lives.\n\n## Script and register, not just language\n\nTwo details specific to how these models are trained, and both matter in South Asia in particular.\n\n**Script changes everything.** Hindi written in Devanagari and Hindi written in Roman letters are, to the tokenizer and often to the model, different problems. A model may be decent at one and poor at the other depending on what its corpus contained: web forums are heavily romanised, formal publishing is not. Test both scripts separately.\n\n**Code-switching is normal and is tested by nobody.** Real Hinglish mixes English words into Hindi grammar, in Roman script, with English technical vocabulary. Models handle this unevenly and no public benchmark covers it. If your users write that way, your evaluation set must too, and it will not resemble anything you can download.\n\n**Register.** A model may produce grammatically correct Hindi that reads as translated newspaper prose to somebody who would have written it conversationally. This is invisible to automated metrics and obvious to a native speaker in one sentence.\n\n## The translation sandwich\n\nA common strategy: translate the input to English, do the work, translate back. It works, and it has three costs worth naming.\n\n- **Two extra model calls** of latency and expense.\n- **Culture-specific content degrades.** Names, honorifics, local references, legal and administrative terms with no English equivalent.\n- **Errors compound across three stages**, and the failure appears in the final output where it is hardest to attribute.\n\nIt remains the right answer when the task is language-neutral (classification into English labels, extracting a date) and the wrong answer when the output goes to a user in their language.\n\nA better version when you can afford it: do the work in the original language with a model that supports it, and use translation only to let *you* check the output.\n\n## Measuring, since benchmarks will not\n\nNo public benchmark will tell you whether a model is good enough for your users in Marathi. The procedure that will, and it takes about two hours:\n\n1. Collect **30 real inputs** in each language you care about, in the script your users actually use.\n2. Write what a good response looks like, or a one-line rubric.\n3. Run three candidate models at temperature 0.\n4. Have a **native speaker** rate each output on three points: acceptable, awkward but usable, wrong.\n5. Count tokens per input for each model while you are there.\n\nThe token count is the cost decision and the ratings are the quality decision, and neither is available anywhere else. Repeat it when you change model, quantisation or provider, because quantisation damages lower-resource languages more than English, as an earlier lesson showed.\n\n## What to reach for when the answer is not good enough\n\n- **A language-focused model.** The families in the earlier lesson exist for this and often win decisively.\n- **Fine-tuning on a few hundred examples in your language and register.** This fixes register and format better than it fixes knowledge, which is usually what is wrong.\n- **Retrieval in the user's language.** Much of what looks like a language weakness is a knowledge weakness, and giving the model the right paragraph in the right language fixes both.\n- **A larger model.** Multilingual quality scales with size more steeply than English quality does, so this helps more here than elsewhere, at the cost the earlier lessons described.\n\n> The habit that matters most: never accept a language claim without a speaker of that language reading twenty outputs. Fluency and correctness diverge sharply outside the first tier, and fluency is what fools a non-speaker."},"check":{"question":"A team evaluating Hindi support finds a model scores well on a public Hindi benchmark but users say the output reads oddly. What is the most likely explanation?","options":[{"text":"The benchmark was contaminated, so the score is meaningless.","correct":false,"tempting":"Possible, but contamination would inflate accuracy rather than produce fluent-but-odd prose that users can read and object to."},{"text":"The model is producing correct but stilted, translation-flavoured Hindi in the wrong register and possibly the wrong script, which automated scoring does not capture.","correct":true},{"text":"Quantisation has damaged the Hindi capability specifically, and the benchmark was run at full precision.","correct":false,"tempting":"A genuine effect worth ruling out, but it would tend to produce errors rather than a consistent register mismatch."},{"text":"The tokenizer is splitting Devanagari inefficiently, which degrades output quality.","correct":false,"tempting":"High fertility raises cost and shortens effective context; it does not by itself produce the wrong register."}]},"prev":{"slug":"agents-on-small-models","title":"Agent loops on models that are not clever","url":"https://addaly.com/learn/open-models/agents-on-small-models"},"next":{"slug":"refusals-and-uncensored-models","title":"Refusals, and the uncensored question","url":"https://addaly.com/learn/open-models/refusals-and-uncensored-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What these models really do in your language', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/multilingual-reality (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What these models really do in your language","courseTitle":"The Open Model Ecosystem","module":{"index":5,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Language support divides into well-supported, thin and nominal tiers with a steep cliff between the last two, script and register matter as much as language, and only a native speaker rating thirty real inputs will tell you which tier you are in.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-tiers-and-where-the-cliff-is","runs":[{"text":"Three tiers, and where the cliff is"}]},{"t":"p","runs":[{"text":"Model cards list languages. The list conflates three very different states."}]},{"t":"p","runs":[{"text":"Well supported.","b":true},{"text":" English, and to varying degrees Chinese, Spanish, French, German, Portuguese and Russian. Fluent, idiomatic, reliable instruction following, reasonable factual coverage."}]},{"t":"p","runs":[{"text":"Present but thinner.","b":true},{"text":" Hindi, Arabic, Bengali, Indonesian, Vietnamese, Turkish, Japanese, Korean and others. Fluent-sounding output with more factual error, weaker instruction adherence, occasional grammatical oddity, and a noticeable drop on anything requiring reasoning in the language rather than translation into it."}]},{"t":"p","runs":[{"text":"Nominal.","b":true},{"text":" Most of the world’s languages. The model produces something, and a native speaker will find it stilted, wrong, or written in the wrong register. The list on the model card is often derived from what appeared in training data at any volume, not from an evaluation."}]},{"t":"p","runs":[{"text":"The cliff between the second and third tier is steep, and it is where most of the disappointment lives."}]},{"t":"h2","id":"script-and-register-not-just-language","runs":[{"text":"Script and register, not just language"}]},{"t":"p","runs":[{"text":"Two details specific to how these models are trained, and both matter in South Asia in particular."}]},{"t":"p","runs":[{"text":"Script changes everything.","b":true},{"text":" Hindi written in Devanagari and Hindi written in Roman letters are, to the tokenizer and often to the model, different problems. A model may be decent at one and poor at the other depending on what its corpus contained: web forums are heavily romanised, formal publishing is not. Test both scripts separately."}]},{"t":"p","runs":[{"text":"Code-switching is normal and is tested by nobody.","b":true},{"text":" Real Hinglish mixes English words into Hindi grammar, in Roman script, with English technical vocabulary. Models handle this unevenly and no public benchmark covers it. If your users write that way, your evaluation set must too, and it will not resemble anything you can download."}]},{"t":"p","runs":[{"text":"Register.","b":true},{"text":" A model may produce grammatically correct Hindi that reads as translated newspaper prose to somebody who would have written it conversationally. This is invisible to automated metrics and obvious to a native speaker in one sentence."}]},{"t":"h2","id":"the-translation-sandwich","runs":[{"text":"The translation sandwich"}]},{"t":"p","runs":[{"text":"A common strategy: translate the input to English, do the work, translate back. It works, and it has three costs worth naming."}]},{"t":"ul","items":[[{"text":"Two extra model calls","b":true},{"text":" of latency and expense."}],[{"text":"Culture-specific content degrades.","b":true},{"text":" Names, honorifics, local references, legal and administrative terms with no English equivalent."}],[{"text":"Errors compound across three stages","b":true},{"text":", and the failure appears in the final output where it is hardest to attribute."}]]},{"t":"p","runs":[{"text":"It remains the right answer when the task is language-neutral (classification into English labels, extracting a date) and the wrong answer when the output goes to a user in their language."}]},{"t":"p","runs":[{"text":"A better version when you can afford it: do the work in the original language with a model that supports it, and use translation only to let "},{"text":"you","i":true},{"text":" check the output."}]},{"t":"h2","id":"measuring-since-benchmarks-will-not","runs":[{"text":"Measuring, since benchmarks will not"}]},{"t":"p","runs":[{"text":"No public benchmark will tell you whether a model is good enough for your users in Marathi. The procedure that will, and it takes about two hours:"}]},{"t":"ol","start":1,"items":[[{"text":"Collect "},{"text":"30 real inputs","b":true},{"text":" in each language you care about, in the script your users actually use."}],[{"text":"Write what a good response looks like, or a one-line rubric."}],[{"text":"Run three candidate models at temperature 0."}],[{"text":"Have a "},{"text":"native speaker","b":true},{"text":" rate each output on three points: acceptable, awkward but usable, wrong."}],[{"text":"Count tokens per input for each model while you are there."}]]},{"t":"p","runs":[{"text":"The token count is the cost decision and the ratings are the quality decision, and neither is available anywhere else. Repeat it when you change model, quantisation or provider, because quantisation damages lower-resource languages more than English, as an earlier lesson showed."}]},{"t":"h2","id":"what-to-reach-for-when-the-answer-is-not-good-enough","runs":[{"text":"What to reach for when the answer is not good enough"}]},{"t":"ul","items":[[{"text":"A language-focused model.","b":true},{"text":" The families in the earlier lesson exist for this and often win decisively."}],[{"text":"Fine-tuning on a few hundred examples in your language and register.","b":true},{"text":" This fixes register and format better than it fixes knowledge, which is usually what is wrong."}],[{"text":"Retrieval in the user’s language.","b":true},{"text":" Much of what looks like a language weakness is a knowledge weakness, and giving the model the right paragraph in the right language fixes both."}],[{"text":"A larger model.","b":true},{"text":" Multilingual quality scales with size more steeply than English quality does, so this helps more here than elsewhere, at the cost the earlier lessons described."}]]},{"t":"quote","runs":[{"text":"The habit that matters most: never accept a language claim without a speaker of that language reading twenty outputs. Fluency and correctness diverge sharply outside the first tier, and fluency is what fools a non-speaker."}]}],"check":{"question":[{"text":"A team evaluating Hindi support finds a model scores well on a public Hindi benchmark but users say the output reads oddly. What is the most likely explanation?"}],"options":[{"text":[{"text":"The benchmark was contaminated, so the score is meaningless."}],"correct":false,"tempting":[{"text":"Possible, but contamination would inflate accuracy rather than produce fluent-but-odd prose that users can read and object to."}]},{"text":[{"text":"The model is producing correct but stilted, translation-flavoured Hindi in the wrong register and possibly the wrong script, which automated scoring does not capture."}],"correct":true,"tempting":null},{"text":[{"text":"Quantisation has damaged the Hindi capability specifically, and the benchmark was run at full precision."}],"correct":false,"tempting":[{"text":"A genuine effect worth ruling out, but it would tend to produce errors rather than a consistent register mismatch."}]},{"text":[{"text":"The tokenizer is splitting Devanagari inefficiently, which degrades output quality."}],"correct":false,"tempting":[{"text":"High fertility raises cost and shortens effective context; it does not by itself produce the wrong register."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"agents-on-small-models","title":"Agent loops on models that are not clever","minutes":9},"next":{"slug":"refusals-and-uncensored-models","title":"Refusals, and the uncensored question","minutes":9}}}