Seven things it will fail at, and why

A boundary you can reason about

Lists of AI limitations are usually just lists. This one attaches a mechanism to each item, because a mechanism lets you predict the next case rather than memorising this one.

1. Arithmetic over more than a few numbers

Adding a column of 300 figures is not a language problem, and next-token prediction produces a number that looks right at about the correct magnitude. There is no carry, no column, no check.

The exception matters: several tools now write and run real code to compute, which is different: the model writes a sum and a computer performs it. If your tool shows you the code it ran, the answer is as good as the code. If it just produces a number, do not trust it. Either way, the total still gets checked against your bank balance.

2. Anything recent

Training stopped on a date. Ask about a rule change from three months ago and you get a fluent account of the old rule, delivered with no hint that time has passed. Web search, where available and switched on, fixes most of this, and you should know whether it is on, because the interface rarely says.

3. Your private facts

Your prices, your lead times, your staff rota, what your supplier promised. None of it was in the training data. Attaching your documents fixes this properly; hoping does not.

4. Exact recall inside a long document

This one surprises people, because it feels like the machine can obviously read. Give it a 90-page contract and ask what clause 14.3 says and it will often be right. Ask it to list every deadline mentioned and it will miss some.

The reason is that attention across a very long input is diluted; material in the middle of a long document is reliably weaker than material at the start or the end. Researchers call it the lost-in-the-middle effect. The practical fix is to work in sections rather than asking one question of the whole thing, and to ask for quotes with page numbers so you can verify what it found.

5. Counting and ordering

“How many times does this appear?” “Put these 60 items in order.” “Which is the third longest?” These require holding state and iterating, which is not what generating likely text does. Use a spreadsheet. Sorting is a solved problem and has been since the 1950s.

6. Constraint problems

A staff rota where Ana cannot work Tuesdays, two people must be on every shift, nobody does more than five days, and Miguel and Sofia cannot be paired. The model produces a rota that looks like a rota and breaks two rules, because it is writing a plausible schedule rather than searching for a valid one.

Rotas, delivery routes and packing are search problems. Real solvers exist and several are free: the scheduling features in booking software, spreadsheet solvers, or open tools like OR-Tools. Ask the model to help you write down the constraints, then give them to something that can actually satisfy them.

7. Anything that needs somebody accountable

A filed tax return. A diagnosis. A structural sign-off. Legal advice about your specific situation. These are not hard for technical reasons; they are hard because when they are wrong, a named person answers for it. A model cannot be that person, cannot be insured, and cannot be struck off.

Use it to prepare: to understand the letter, to draft the questions, to organise your papers before an appointment. Never to decide.

The pattern under all seven

Six of these are the same failure wearing different clothes: the machine produces something with the shape of a correct answer without any process that could make it correct. A rota that looks like a rota. A total that looks like a total. A clause list that looks complete.

Which gives you a test you can apply to any new task in about five seconds. Would a wrong answer look different from a right one? If yes, you are probably fine. If a wrong answer would look identical, you need a source, a calculation or a person, and you need it before the output reaches anybody.