{"course":{"slug":"ai-for-teachers","title":"AI for Teachers","url":"https://addaly.com/learn/ai-for-teachers","level":"beginner","track":"work"},"lesson":{"slug":"a-chatbot-is-not-a-calculator","title":"A chatbot is not a calculator","url":"https://addaly.com/learn/ai-for-teachers/a-chatbot-is-not-a-calculator","position":7,"of":57,"minutes":8,"answer":"Next-word prediction is a poor way to multiply, so every number in generated material is checked with a calculator.","keyIdea":"Next-word prediction is a poor way to multiply, so every number in generated material is checked with a calculator, and the safe pattern is to have the model write the working while a separate tool does the arithmetic."},"body":{"format":"markdown","text":"## Why 47 × 83 is hard for it and easy for a two-rupee calculator\n\nA calculator computes. It has a circuit that multiplies and the answer is exact every time.\n\nA language model predicts. When it writes \"47 × 83 = 3901\", it is producing the digits that most commonly followed similar-looking expressions in text. For small, common products (anything in the times tables, round numbers) it has seen the answer so often that it is effectively memorised. For 47 × 83 it has seen the exact expression rarely, and it assembles the digits from pattern. It is right more often than chance and wrong often enough that you cannot trust it. The correct answer, incidentally, is 3901; try a few of your own and watch it slip on larger ones.\n\nThis is the same mechanism as every other lesson in this block. Arithmetic is a specific with few occurrences in text. That is the unreliable list.\n\n## What it gets wrong, specifically\n\n- **Multi-digit multiplication and division.** Increasingly wrong as the numbers grow.\n- **Long chains.** A five-step calculation where step three is slightly off, carried through with perfect confidence.\n- **Counting.** How many words in this paragraph, how many items in this list. It does not count; it estimates.\n- **Unit conversion with awkward factors.** 1 acre to square metres, done from pattern.\n- **Anything with a carry across many columns.**\n\nWhat it gets right is the method. The setting-up of an equation, the choice of formula, the order of operations, the explanation of why: these are language, and language is what it does. Which produces the pattern in the check question: sound working, wrong number.\n\n## The safe division of labour\n\nLet the model do what it is good at and hand the numbers to something that computes.\n\n**Generating problems.** Ask for the problems and the method, and produce the answer key yourself with a calculator. Ten problems take five minutes to check. Or ask for problems whose answers you specify: \"Write a word problem about a shopkeeper whose answer is exactly 240 rupees.\" It will fit the story to the number, which is far more reliable than fitting a number to the story.\n\n> Figure: Where the arithmetic falls apart. Bars (% of tries with the correct answer): 7 x 8 100; 23 x 4 97; 47 x 83 71; 382 x 47 38; 5184 divided by 24 29. One free-tier chatbot, each expression asked a hundred times in fresh chats, no calculator and no code. Small common products are effectively memorised. Accuracy falls as the digits grow, because the exact expression appears in text less and less often. The method in the working stayed sound throughout, which is exactly what makes the wrong numbers hard to notice.\n\n**Checking student working.** \"Here is a student's solution. Identify the step where the method goes wrong.\" It is good at this. Then verify the arithmetic yourself.\n\n**Code, if you have it.** Several tools can write and run a short piece of Python when asked, and Python computes exactly. \"Solve this using code and show the result\" produces a real answer. Free tiers include this in some products and not others. If you see a snippet of code and its output in the answer, the number came from a computation. If you see only prose, it came from prediction.\n\n**\"Reasoning\" modes.** Newer models have a mode that works through problems in steps before answering, and it markedly improves arithmetic: the model effectively writes out a long-multiplication and reads the answer off its own working. It is slower and often limited on free tiers. Even then, verify. Better is not the same as exact.\n\n## A classroom use that turns the weakness into a lesson\n\nAsk the model, in front of the class or on a printed transcript, to multiply two three-digit numbers. Have students check it on paper. When it is wrong, and choose numbers where it is, the lesson lands: this machine is fluent and it cannot multiply. Then ask why. A Class 8 student who has done the prediction activity from the first lesson can answer: because it guesses digits the way it guesses words.\n\n## The limitation\n\nThe line is moving. Each generation of models does arithmetic better, and tools that quietly call a calculator behind the scenes are becoming common. So you may find your tool gets 47 × 83 right every time you try. That does not change the rule, because you cannot see from the outside whether a given answer came from a computation or a guess. Check the numbers. It costs a minute and it is the minute that stops a worksheet with a wrong answer key reaching forty children."},"check":{"question":"A maths teacher asks a chatbot to produce ten word problems with an answer key. She notices the model's step-by-step working on question 6 is sound but the final product is off by 40. What best explains this?","options":[{"text":"The model made a deliberate small error to test whether the teacher was actually checking the key before printing.","correct":false,"tempting":"Attributes intent; the model has no goals about her behaviour and no mechanism to plant a test."},{"text":"The word problem was badly worded, so the model solved a slightly different problem from the one it had written a moment before.","correct":false,"tempting":"Does not fit the evidence; the working matched the problem and only the final arithmetic failed."},{"text":"The method is language, which it handles well; the product is digits predicted from pattern, which fails on uncommon combinations.","correct":true},{"text":"The answer key was generated in a separate pass from the questions, so the number belongs to a different question in a different set.","correct":false,"tempting":"Possible in some workflows but does not explain a single wrong product inside otherwise correct working on the same question."}]},"prev":{"slug":"it-has-read-the-american-curriculum","title":"It has read the American curriculum","url":"https://addaly.com/learn/ai-for-teachers/it-has-read-the-american-curriculum"},"next":{"slug":"same-question-different-answer","title":"Same question, different answer","url":"https://addaly.com/learn/ai-for-teachers/same-question-different-answer"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A chatbot is not a calculator', in AI for Teachers. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-teachers/a-chatbot-is-not-a-calculator (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A chatbot is not a calculator","courseTitle":"AI for Teachers","module":{"index":1,"slug":"understanding-the-machine","title":"Understanding the machine you are about to use"},"answer":null,"keyIdea":"Next-word prediction is a poor way to multiply, so every number in generated material is checked with a calculator, and the safe pattern is to have the model write the working while a separate tool does the arithmetic.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of tries with the correct answer","title":"Where the arithmetic falls apart","series":[{"label":"7 x 8","value":100},{"label":"23 x 4","value":97},{"label":"47 x 83","value":71,"highlight":true},{"label":"382 x 47","value":38},{"label":"5184 divided by 24","value":29}],"caption":"One free-tier chatbot, each expression asked a hundred times in fresh chats, no calculator and no code. Small common products are effectively memorised. Accuracy falls as the digits grow, because the exact expression appears in text less and less often. The method in the working stayed sound throughout, which is exactly what makes the wrong numbers hard to notice."},"alt":"Figure: Where the arithmetic falls apart. Bars (% of tries with the correct answer): 7 x 8 100; 23 x 4 97; 47 x 83 71; 382 x 47 38; 5184 divided by 24 29. One free-tier chatbot, each expression asked a hundred times in fresh chats, no calculator and no code. Small common products are effectively memorised. Accuracy falls as the digits grow, because the exact expression appears in text less and less often. The method in the working stayed sound throughout, which is exactly what makes the wrong numbers hard to notice."}},"blocks":[{"t":"h2","id":"why-47-83-is-hard-for-it-and-easy-for-a-two-rupee-calculator","runs":[{"text":"Why 47 × 83 is hard for it and easy for a two-rupee calculator"}]},{"t":"p","runs":[{"text":"A calculator computes. It has a circuit that multiplies and the answer is exact every time."}]},{"t":"p","runs":[{"text":"A language model predicts. When it writes “47 × 83 = 3901”, it is producing the digits that most commonly followed similar-looking expressions in text. For small, common products (anything in the times tables, round numbers) it has seen the answer so often that it is effectively memorised. For 47 × 83 it has seen the exact expression rarely, and it assembles the digits from pattern. It is right more often than chance and wrong often enough that you cannot trust it. The correct answer, incidentally, is 3901; try a few of your own and watch it slip on larger ones."}]},{"t":"p","runs":[{"text":"This is the same mechanism as every other lesson in this block. Arithmetic is a specific with few occurrences in text. That is the unreliable list."}]},{"t":"h2","id":"what-it-gets-wrong-specifically","runs":[{"text":"What it gets wrong, specifically"}]},{"t":"ul","items":[[{"text":"Multi-digit multiplication and division.","b":true},{"text":" Increasingly wrong as the numbers grow."}],[{"text":"Long chains.","b":true},{"text":" A five-step calculation where step three is slightly off, carried through with perfect confidence."}],[{"text":"Counting.","b":true},{"text":" How many words in this paragraph, how many items in this list. It does not count; it estimates."}],[{"text":"Unit conversion with awkward factors.","b":true},{"text":" 1 acre to square metres, done from pattern."}],[{"text":"Anything with a carry across many columns.","b":true}]]},{"t":"p","runs":[{"text":"What it gets right is the method. The setting-up of an equation, the choice of formula, the order of operations, the explanation of why: these are language, and language is what it does. Which produces the pattern in the check question: sound working, wrong number."}]},{"t":"h2","id":"the-safe-division-of-labour","runs":[{"text":"The safe division of labour"}]},{"t":"p","runs":[{"text":"Let the model do what it is good at and hand the numbers to something that computes."}]},{"t":"p","runs":[{"text":"Generating problems.","b":true},{"text":" Ask for the problems and the method, and produce the answer key yourself with a calculator. Ten problems take five minutes to check. Or ask for problems whose answers you specify: “Write a word problem about a shopkeeper whose answer is exactly 240 rupees.” It will fit the story to the number, which is far more reliable than fitting a number to the story."}]},{"t":"p","runs":[{"text":"Checking student working.","b":true},{"text":" “Here is a student’s solution. Identify the step where the method goes wrong.” It is good at this. Then verify the arithmetic yourself."}]},{"t":"p","runs":[{"text":"Code, if you have it.","b":true},{"text":" Several tools can write and run a short piece of Python when asked, and Python computes exactly. “Solve this using code and show the result” produces a real answer. Free tiers include this in some products and not others. If you see a snippet of code and its output in the answer, the number came from a computation. If you see only prose, it came from prediction."}]},{"t":"p","runs":[{"text":"”Reasoning” modes.","b":true},{"text":" Newer models have a mode that works through problems in steps before answering, and it markedly improves arithmetic: the model effectively writes out a long-multiplication and reads the answer off its own working. It is slower and often limited on free tiers. Even then, verify. Better is not the same as exact."}]},{"t":"h2","id":"a-classroom-use-that-turns-the-weakness-into-a-lesson","runs":[{"text":"A classroom use that turns the weakness into a lesson"}]},{"t":"p","runs":[{"text":"Ask the model, in front of the class or on a printed transcript, to multiply two three-digit numbers. Have students check it on paper. When it is wrong, and choose numbers where it is, the lesson lands: this machine is fluent and it cannot multiply. Then ask why. A Class 8 student who has done the prediction activity from the first lesson can answer: because it guesses digits the way it guesses words."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"The line is moving. Each generation of models does arithmetic better, and tools that quietly call a calculator behind the scenes are becoming common. So you may find your tool gets 47 × 83 right every time you try. That does not change the rule, because you cannot see from the outside whether a given answer came from a computation or a guess. Check the numbers. It costs a minute and it is the minute that stops a worksheet with a wrong answer key reaching forty children."}]}],"check":{"question":[{"text":"A maths teacher asks a chatbot to produce ten word problems with an answer key. She notices the model’s step-by-step working on question 6 is sound but the final product is off by 40. What best explains this?"}],"options":[{"text":[{"text":"The model made a deliberate small error to test whether the teacher was actually checking the key before printing."}],"correct":false,"tempting":[{"text":"Attributes intent; the model has no goals about her behaviour and no mechanism to plant a test."}]},{"text":[{"text":"The word problem was badly worded, so the model solved a slightly different problem from the one it had written a moment before."}],"correct":false,"tempting":[{"text":"Does not fit the evidence; the working matched the problem and only the final arithmetic failed."}]},{"text":[{"text":"The method is language, which it handles well; the product is digits predicted from pattern, which fails on uncommon combinations."}],"correct":true,"tempting":null},{"text":[{"text":"The answer key was generated in a separate pass from the questions, so the number belongs to a different question in a different set."}],"correct":false,"tempting":[{"text":"Possible in some workflows but does not explain a single wrong product inside otherwise correct working on the same question."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"it-has-read-the-american-curriculum","title":"It has read the American curriculum","minutes":8},"next":{"slug":"same-question-different-answer","title":"Same question, different answer","minutes":8}}}