{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"real-differences","title":"Where they differ, and where it is marketing","url":"https://addaly.com/learn/choosing-your-tools/real-differences","position":10,"of":76,"minutes":8,"answer":"Your five real tasks tell you more than any leaderboard, because a benchmark measures the test and not your work.","keyIdea":"Your five real tasks tell you more than any leaderboard, because a benchmark measures the test and not your work."},"body":{"format":"markdown","text":"## Differences you will actually feel\n\nSome gaps between these tools are real and show up in an afternoon of use.\n\n**Language coverage.** This is the biggest under-advertised difference. Models are trained on wildly different mixtures of text, and quality in Marathi, Yoruba, Vietnamese or Quechua varies far more between models than quality in English does. Nobody puts this on a comparison page. Test it yourself in the language you actually work in.\n\n**What the product can reach.** Can it read your Google Drive? Run code and show you the output? Live inside your editor and see the whole project? Browse the web and cite what it found? These are product decisions, and they differ enormously. For most jobs this matters more than model quality.\n\n**Long documents.** Advertised context windows range from tens of thousands of tokens to a million or more. But a stated window is a ceiling, not a promise of quality. Some models hold detail from page 200 well and some lose the middle. Test it: paste a long document, then ask about something buried two-thirds of the way in.\n\n**House manners.** Models are trained with different preferences and it shows. One hedges heavily and asks clarifying questions. Another commits to an answer and is wrong with confidence. One refuses a medical question, another answers with caveats. This is real, persistent, and a matter of taste, but it decides whether a tool fits your work.\n\n**Availability and payment.** In some countries an account is hard to create, a card is hard to charge, or the service is not offered. This ends the debate quickly.\n\n**Price per unit of work.** Invisible when you are chatting. Decisive when you are processing 50,000 support tickets, where a tenfold cost difference between a small model and a frontier one is the whole budget.\n\n## Differences that are mostly marketing\n\n**Leaderboard positions two or three points apart.** Public benchmarks are useful for tracking the field over years, and close to useless for choosing between today's top few. The tests get saturated, questions leak into training data, and scores are reported under favourable settings. A three-point gap on a reasoning benchmark will not be visible in your work.\n\n**\"Reasoning\" as a brand word.** Most serious models now have a mode where they spend longer thinking before answering, and it helps on maths, logic and multi-step planning. It is not a separate species of intelligence that one company owns. Where they differ is how much compute they spend and whether you can control it.\n\n**Parameter counts.** \"Trillions of parameters\" is unverifiable for closed models and does not map cleanly to usefulness. A well-trained 30-billion-parameter model beats a badly-trained 200-billion one.\n\n**Launch demos.** Every one of them is a best case, rehearsed, recorded, and sometimes edited. Treat them as advertising, because they are.\n\n## Build your own benchmark in twenty minutes\n\nHere is the method that beats every review you will read.\n\nOpen a plain text file. Write down five tasks you do, with real material:\n\n1. A real email you need to reply to, with the context you would give a colleague.\n2. A document from your work, plus the question you would actually ask about it.\n3. A task in your own language, not English.\n4. Something you already know the correct answer to. This is the one that catches confident nonsense.\n5. The thing you were doing when you last got frustrated with a tool.\n\nRun all five through each candidate. Judge them yourself. It takes less than an hour and it is worth more than every comparison table on the internet, because it measures fit to your work rather than fit to a test.\n\nKeep the file. You will use it again in lesson eight, when everything has changed and you need to decide all over again."},"check":{"question":"Two assistants sit three points apart on a public reasoning benchmark. Ravi picks the higher one for summarizing Marathi news. What is wrong with his reasoning?","options":[{"text":"Nothing is wrong. A benchmark is the only objective measure available, so a three-point gap is the best evidence he has.","correct":false,"tempting":"Objective-looking is not the same as relevant; a measurement only helps if the test resembles the job."},{"text":"He should have picked the lower-scoring one, since leaderboard leaders are usually tuned for tests rather than real work.","correct":false,"tempting":"Test-tuning is real, but inverting the leaderboard is superstition pointed the other way."},{"text":"The benchmark measures English reasoning problems and says nothing about how either model handles Marathi.","correct":true},{"text":"Benchmarks are meaningless because the scores change every few months anyway.","correct":false,"tempting":"They change because the models change; the problem here is fit to his task, not instability."}]},"prev":{"slug":"under-new-management","title":"When the engine gets swapped","url":"https://addaly.com/learn/choosing-your-tools/under-new-management"},"next":{"slug":"the-prompt-file","title":"The file you will keep for years","url":"https://addaly.com/learn/choosing-your-tools/the-prompt-file"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where they differ, and where it is marketing', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/real-differences (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where they differ, and where it is marketing","courseTitle":"Choosing and Using the Tools","module":{"index":2,"slug":"testing-it-yourself","title":"Testing a tool yourself"},"answer":null,"keyIdea":"Your five real tasks tell you more than any leaderboard, because a benchmark measures the test and not your work.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"differences-you-will-actually-feel","runs":[{"text":"Differences you will actually feel"}]},{"t":"p","runs":[{"text":"Some gaps between these tools are real and show up in an afternoon of use."}]},{"t":"p","runs":[{"text":"Language coverage.","b":true},{"text":" This is the biggest under-advertised difference. Models are trained on wildly different mixtures of text, and quality in Marathi, Yoruba, Vietnamese or Quechua varies far more between models than quality in English does. Nobody puts this on a comparison page. Test it yourself in the language you actually work in."}]},{"t":"p","runs":[{"text":"What the product can reach.","b":true},{"text":" Can it read your Google Drive? Run code and show you the output? Live inside your editor and see the whole project? Browse the web and cite what it found? These are product decisions, and they differ enormously. For most jobs this matters more than model quality."}]},{"t":"p","runs":[{"text":"Long documents.","b":true},{"text":" Advertised context windows range from tens of thousands of tokens to a million or more. But a stated window is a ceiling, not a promise of quality. Some models hold detail from page 200 well and some lose the middle. Test it: paste a long document, then ask about something buried two-thirds of the way in."}]},{"t":"p","runs":[{"text":"House manners.","b":true},{"text":" Models are trained with different preferences and it shows. One hedges heavily and asks clarifying questions. Another commits to an answer and is wrong with confidence. One refuses a medical question, another answers with caveats. This is real, persistent, and a matter of taste, but it decides whether a tool fits your work."}]},{"t":"p","runs":[{"text":"Availability and payment.","b":true},{"text":" In some countries an account is hard to create, a card is hard to charge, or the service is not offered. This ends the debate quickly."}]},{"t":"p","runs":[{"text":"Price per unit of work.","b":true},{"text":" Invisible when you are chatting. Decisive when you are processing 50,000 support tickets, where a tenfold cost difference between a small model and a frontier one is the whole budget."}]},{"t":"h2","id":"differences-that-are-mostly-marketing","runs":[{"text":"Differences that are mostly marketing"}]},{"t":"p","runs":[{"text":"Leaderboard positions two or three points apart.","b":true},{"text":" Public benchmarks are useful for tracking the field over years, and close to useless for choosing between today’s top few. The tests get saturated, questions leak into training data, and scores are reported under favourable settings. A three-point gap on a reasoning benchmark will not be visible in your work."}]},{"t":"p","runs":[{"text":"”Reasoning” as a brand word.","b":true},{"text":" Most serious models now have a mode where they spend longer thinking before answering, and it helps on maths, logic and multi-step planning. It is not a separate species of intelligence that one company owns. Where they differ is how much compute they spend and whether you can control it."}]},{"t":"p","runs":[{"text":"Parameter counts.","b":true},{"text":" “Trillions of parameters” is unverifiable for closed models and does not map cleanly to usefulness. A well-trained 30-billion-parameter model beats a badly-trained 200-billion one."}]},{"t":"p","runs":[{"text":"Launch demos.","b":true},{"text":" Every one of them is a best case, rehearsed, recorded, and sometimes edited. Treat them as advertising, because they are."}]},{"t":"h2","id":"build-your-own-benchmark-in-twenty-minutes","runs":[{"text":"Build your own benchmark in twenty minutes"}]},{"t":"p","runs":[{"text":"Here is the method that beats every review you will read."}]},{"t":"p","runs":[{"text":"Open a plain text file. Write down five tasks you do, with real material:"}]},{"t":"ol","start":1,"items":[[{"text":"A real email you need to reply to, with the context you would give a colleague."}],[{"text":"A document from your work, plus the question you would actually ask about it."}],[{"text":"A task in your own language, not English."}],[{"text":"Something you already know the correct answer to. This is the one that catches confident nonsense."}],[{"text":"The thing you were doing when you last got frustrated with a tool."}]]},{"t":"p","runs":[{"text":"Run all five through each candidate. Judge them yourself. It takes less than an hour and it is worth more than every comparison table on the internet, because it measures fit to your work rather than fit to a test."}]},{"t":"p","runs":[{"text":"Keep the file. You will use it again in lesson eight, when everything has changed and you need to decide all over again."}]}],"check":{"question":[{"text":"Two assistants sit three points apart on a public reasoning benchmark. Ravi picks the higher one for summarizing Marathi news. What is wrong with his reasoning?"}],"options":[{"text":[{"text":"Nothing is wrong. A benchmark is the only objective measure available, so a three-point gap is the best evidence he has."}],"correct":false,"tempting":[{"text":"Objective-looking is not the same as relevant; a measurement only helps if the test resembles the job."}]},{"text":[{"text":"He should have picked the lower-scoring one, since leaderboard leaders are usually tuned for tests rather than real work."}],"correct":false,"tempting":[{"text":"Test-tuning is real, but inverting the leaderboard is superstition pointed the other way."}]},{"text":[{"text":"The benchmark measures English reasoning problems and says nothing about how either model handles Marathi."}],"correct":true,"tempting":null},{"text":[{"text":"Benchmarks are meaningless because the scores change every few months anyway."}],"correct":false,"tempting":[{"text":"They change because the models change; the problem here is fit to his task, not instability."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"under-new-management","title":"When the engine gets swapped","minutes":8},"next":{"slug":"the-prompt-file","title":"The file you will keep for years","minutes":9}}}