Arithmetic it should not do in its head

Why the sums go wrong

A model produces digits the way it produces words: one token at a time, each chosen because it is a likely continuation. There is no arithmetic unit. On small, familiar sums this works, because the patterns are all over the training data. On long multiplication, percentages of awkward numbers, date arithmetic across months, or a column of forty figures, it drifts.

The failure is quiet. 4,182 × 37 produces a plausible six-digit number that is wrong in the middle. Nothing about the output looks different from a correct one.

Add the tokenisation problem from module one (long numbers are split into digit groups in ways that vary between models) and you have a system that is fluent about quantities and unreliable with them.

The fix: make it write code, not answers

Most current chat products can execute code. ChatGPT, Claude, Gemini and Copilot all offer some form of it, usually Python. Ask for the calculation to be done that way:

Do not calculate this in prose. Write Python that computes it, run it, and show me both the code and the result.

Now the arithmetic is performed by an interpreter that does not guess, and you get something you can read:

rows = [("Mar", 41820, 0.18), ("Apr", 38945, 0.18), ("May", 52310, 0.12)]
total = sum(amount * (1 + vat) for _, amount, vat in rows)
print(round(total, 2))

Three things improved at once. The number is computed rather than generated. The code shows which figures and which rate were used, so a wrong assumption is visible. And you can rerun it next month by changing three lines.

Where this matters most

Anything cumulative. Totals, running balances, compound interest, percentage change across periods. Errors here compound in the ordinary sense as well as the mathematical one.

Anything with dates. “Ninety working days from 14 March, excluding Kenyan public holidays” is a calculation, not a fact, and models are unreliable at it in prose.

Anything over more than about ten rows. Summing a short column in prose is usually fine. Summing forty is not, and the error rate rises exactly where checking gets harder.

Anything a spreadsheet would normally do. Grouping, sorting by two keys, deduplicating, joining two lists on a shared column.

Checking code you cannot write

You do not need to write Python to use this. You need to read four things:

1. The input numbers. Do they match your source? This catches most errors.
2. The operation. Multiplication where you expected addition, a rate of 0.18 where the invoice says 18%.
3. The row count. len(rows) should equal what you gave it. A quiet drop of three rows is the most common silent failure.
4. The result’s plausibility. Roughly what you expected? An order of magnitude out is the sign of a unit error.

Ask for those to be printed: “print the row count and the first two rows before the total”. Thirty seconds of checking, and it catches the class of error that a bare number never reveals.

The free path

If your tool cannot run code, or the data should not leave your machine, you can do this offline. Ask for the Python and run it yourself: Python is free, installed on most Macs and Linux machines, and a small download on Windows. LibreOffice Calc, also free, will do the same work with formulas, and asking for the formula rather than the answer is the same trick in a spreadsheet.

Give me the LibreOffice/Excel formula for this, not the answer. Explain what each part refers to.

A formula sits in the sheet and recalculates. A number pasted from a chat is a fossil that nobody can audit next quarter.

The rule

Ask for the method, not the number, whenever the number matters. A computed result you can rerun beats a generated result you have to trust, and the difference costs you one sentence in the prompt.

What code execution does not fix

Two things, and both catch people who have just discovered this technique.

Wrong inputs. If the model misread 41,820 as 4,182 while pulling figures out of your pasted invoice, the code computes a perfect total of the wrong numbers. Extraction and calculation are separate steps with separate failure modes, which is exactly why printing the parsed rows matters more than checking the total.

Wrong logic. Code that applies VAT to a line that was already inclusive is correct code and a wrong answer. Reading the operation is not optional; it is the check.

The gain here is real and specific: it removes arithmetic error, which was one class of failure among several. It does not turn an unverified answer into a verified one.