A chatbot is not a calculator

Why 47 × 83 is hard for it and easy for a two-rupee calculator

A calculator computes. It has a circuit that multiplies and the answer is exact every time.

A language model predicts. When it writes “47 × 83 = 3901”, it is producing the digits that most commonly followed similar-looking expressions in text. For small, common products (anything in the times tables, round numbers) it has seen the answer so often that it is effectively memorised. For 47 × 83 it has seen the exact expression rarely, and it assembles the digits from pattern. It is right more often than chance and wrong often enough that you cannot trust it. The correct answer, incidentally, is 3901; try a few of your own and watch it slip on larger ones.

This is the same mechanism as every other lesson in this block. Arithmetic is a specific with few occurrences in text. That is the unreliable list.

What it gets wrong, specifically

• Multi-digit multiplication and division. Increasingly wrong as the numbers grow.
• Long chains. A five-step calculation where step three is slightly off, carried through with perfect confidence.
• Counting. How many words in this paragraph, how many items in this list. It does not count; it estimates.
• Unit conversion with awkward factors. 1 acre to square metres, done from pattern.
• Anything with a carry across many columns.

What it gets right is the method. The setting-up of an equation, the choice of formula, the order of operations, the explanation of why: these are language, and language is what it does. Which produces the pattern in the check question: sound working, wrong number.

The safe division of labour

Let the model do what it is good at and hand the numbers to something that computes.

Generating problems. Ask for the problems and the method, and produce the answer key yourself with a calculator. Ten problems take five minutes to check. Or ask for problems whose answers you specify: “Write a word problem about a shopkeeper whose answer is exactly 240 rupees.” It will fit the story to the number, which is far more reliable than fitting a number to the story.

Checking student working. “Here is a student’s solution. Identify the step where the method goes wrong.” It is good at this. Then verify the arithmetic yourself.

Code, if you have it. Several tools can write and run a short piece of Python when asked, and Python computes exactly. “Solve this using code and show the result” produces a real answer. Free tiers include this in some products and not others. If you see a snippet of code and its output in the answer, the number came from a computation. If you see only prose, it came from prediction.

”Reasoning” modes. Newer models have a mode that works through problems in steps before answering, and it markedly improves arithmetic: the model effectively writes out a long-multiplication and reads the answer off its own working. It is slower and often limited on free tiers. Even then, verify. Better is not the same as exact.

A classroom use that turns the weakness into a lesson

Ask the model, in front of the class or on a printed transcript, to multiply two three-digit numbers. Have students check it on paper. When it is wrong, and choose numbers where it is, the lesson lands: this machine is fluent and it cannot multiply. Then ask why. A Class 8 student who has done the prediction activity from the first lesson can answer: because it guesses digits the way it guesses words.

The limitation

The line is moving. Each generation of models does arithmetic better, and tools that quietly call a calculator behind the scenes are becoming common. So you may find your tool gets 47 × 83 right every time you try. That does not change the rule, because you cannot see from the outside whether a given answer came from a computation or a guess. Check the numbers. It costs a minute and it is the minute that stops a worksheet with a wrong answer key reaching forty children.