{"course":{"slug":"ai-for-small-business","title":"AI for a Small Business","url":"https://addaly.com/learn/ai-for-small-business","level":"beginner","track":"work"},"lesson":{"slug":"jobs-it-cannot-do","title":"Seven things it will fail at, and why","url":"https://addaly.com/learn/ai-for-small-business/jobs-it-cannot-do","position":7,"of":88,"minutes":9,"answer":"The model fails wherever producing text that has the shape of a correct answer is not the same as computing one (arithmetic, rotas, exhaustive lists.","keyIdea":"The model fails wherever producing text that has the shape of a correct answer is not the same as computing one (arithmetic, rotas, exhaustive lists, counting) and the test is whether a wrong answer would look any different from a right one."},"body":{"format":"markdown","text":"## A boundary you can reason about\n\nLists of AI limitations are usually just lists. This one attaches a mechanism to each item, because a mechanism lets you predict the next case rather than memorising this one.\n\n### 1. Arithmetic over more than a few numbers\n\nAdding a column of 300 figures is not a language problem, and next-token prediction produces a number that looks right at about the correct magnitude. There is no carry, no column, no check.\n\nThe exception matters: several tools now write and run real code to compute, which is different: the model writes a sum and a computer performs it. If your tool shows you the code it ran, the answer is as good as the code. If it just produces a number, do not trust it. Either way, the total still gets checked against your bank balance.\n\n### 2. Anything recent\n\nTraining stopped on a date. Ask about a rule change from three months ago and you get a fluent account of the old rule, delivered with no hint that time has passed. Web search, where available and switched on, fixes most of this, and you should know whether it is on, because the interface rarely says.\n\n### 3. Your private facts\n\nYour prices, your lead times, your staff rota, what your supplier promised. None of it was in the training data. Attaching your documents fixes this properly; hoping does not.\n\n### 4. Exact recall inside a long document\n\nThis one surprises people, because it feels like the machine can obviously read. Give it a 90-page contract and ask what clause 14.3 says and it will often be right. Ask it to list every deadline mentioned and it will miss some.\n\nThe reason is that attention across a very long input is diluted; material in the middle of a long document is reliably weaker than material at the start or the end. Researchers call it the lost-in-the-middle effect. The practical fix is to work in sections rather than asking one question of the whole thing, and to ask for quotes with page numbers so you can verify what it found.\n\n### 5. Counting and ordering\n\n\"How many times does this appear?\" \"Put these 60 items in order.\" \"Which is the third longest?\" These require holding state and iterating, which is not what generating likely text does. Use a spreadsheet. Sorting is a solved problem and has been since the 1950s.\n\n### 6. Constraint problems\n\nA staff rota where Ana cannot work Tuesdays, two people must be on every shift, nobody does more than five days, and Miguel and Sofia cannot be paired. The model produces a rota that looks like a rota and breaks two rules, because it is writing a plausible schedule rather than searching for a valid one.\n\nRotas, delivery routes and packing are search problems. Real solvers exist and several are free: the scheduling features in booking software, spreadsheet solvers, or open tools like OR-Tools. Ask the model to help you write down the constraints, then give them to something that can actually satisfy them.\n\n### 7. Anything that needs somebody accountable\n\nA filed tax return. A diagnosis. A structural sign-off. Legal advice about your specific situation. These are not hard for technical reasons; they are hard because when they are wrong, a named person answers for it. A model cannot be that person, cannot be insured, and cannot be struck off.\n\nUse it to prepare: to understand the letter, to draft the questions, to organise your papers before an appointment. Never to decide.\n\n## The pattern under all seven\n\nSix of these are the same failure wearing different clothes: the machine produces something with the **shape** of a correct answer without any process that could make it correct. A rota that looks like a rota. A total that looks like a total. A clause list that looks complete.\n\n> Figure: The five-second test for any new task. A wrong answer would look wrong: Rewriting an angry reply calmly; Sorting 300 transaction descriptions; Turning your notes into a scope section; Forty descriptions in the pattern of five; Explaining what a clause means. A wrong answer would look right: The total of a column of 300 figures; Every deadline in a 90-page contract; A rota that satisfies all six rules; This year's threshold in your country; Your lead times and your prices. Six of the seven failures are one failure wearing different clothes: text with the shape of a correct answer, produced by nothing that could make it correct. If a wrong answer would be indistinguishable, you need a source, a calculation or a person before the output reaches anybody.\n\nWhich gives you a test you can apply to any new task in about five seconds. **Would a wrong answer look different from a right one?** If yes, you are probably fine. If a wrong answer would look identical, you need a source, a calculation or a person, and you need it before the output reaches anybody."},"check":{"question":"A restaurant asks a model to build next week's staff rota from a list of eight constraints. The rota it returns is neatly formatted, covers every shift, and breaks two of the constraints. Why did this happen?","options":[{"text":"The constraints were not stated clearly enough, so restating them more precisely would produce a valid rota.","correct":false,"tempting":"Assumes the failure is comprehension; the model can restate all eight constraints correctly when asked, and still generate a rota that violates them."},{"text":"Rota-building is a search for an arrangement that satisfies every rule at once, and generating plausible text never performs that search, so a rota-shaped output is not a checked one.","correct":true},{"text":"Eight constraints exceed what the model can hold at once, so the earlier ones were forgotten by the time it reached the end of the week and began filling in the remaining shifts.","correct":false,"tempting":"Blames context length for a reasoning failure; eight short rules occupy a trivial number of tokens and the same failure appears with three."},{"text":"The model prioritised covering every shift over the individual constraints, which is a reasonable trade-off for it to have made on its own given that an uncovered shift closes the restaurant.","correct":false,"tempting":"Attributes deliberate prioritisation to a process that does not weigh objectives; nothing in generation evaluates coverage against constraints or trades one for the other."}]},"prev":{"slug":"the-five-day-tally","title":"The five-day tally, and why memory is not a baseline","url":"https://addaly.com/learn/ai-for-small-business/the-five-day-tally"},"next":{"slug":"one-owner-one-tool","title":"One person owns it, for one month","url":"https://addaly.com/learn/ai-for-small-business/one-owner-one-tool"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Seven things it will fail at, and why', in AI for a Small Business. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-small-business/jobs-it-cannot-do (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Seven things it will fail at, and why","courseTitle":"AI for a Small Business","module":{"index":1,"slug":"where-it-belongs","title":"Working out where AI belongs in your business"},"answer":null,"keyIdea":"The model fails wherever producing text that has the shape of a correct answer is not the same as computing one (arithmetic, rotas, exhaustive lists, counting) and the test is whether a wrong answer would look any different from a right one.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"A wrong answer would look wrong","items":["Rewriting an angry reply calmly","Sorting 300 transaction descriptions","Turning your notes into a scope section","Forty descriptions in the pattern of five","Explaining what a clause means"]},"type":"compare","right":{"title":"A wrong answer would look right","items":["The total of a column of 300 figures","Every deadline in a 90-page contract","A rota that satisfies all six rules","This year’s threshold in your country","Your lead times and your prices"]},"title":"The five-second test for any new task","caption":"Six of the seven failures are one failure wearing different clothes: text with the shape of a correct answer, produced by nothing that could make it correct. If a wrong answer would be indistinguishable, you need a source, a calculation or a person before the output reaches anybody."},"alt":"Figure: The five-second test for any new task. A wrong answer would look wrong: Rewriting an angry reply calmly; Sorting 300 transaction descriptions; Turning your notes into a scope section; Forty descriptions in the pattern of five; Explaining what a clause means. A wrong answer would look right: The total of a column of 300 figures; Every deadline in a 90-page contract; A rota that satisfies all six rules; This year’s threshold in your country; Your lead times and your prices. Six of the seven failures are one failure wearing different clothes: text with the shape of a correct answer, produced by nothing that could make it correct. If a wrong answer would be indistinguishable, you need a source, a calculation or a person before the output reaches anybody."}},"blocks":[{"t":"h2","id":"a-boundary-you-can-reason-about","runs":[{"text":"A boundary you can reason about"}]},{"t":"p","runs":[{"text":"Lists of AI limitations are usually just lists. This one attaches a mechanism to each item, because a mechanism lets you predict the next case rather than memorising this one."}]},{"t":"h3","id":"1-arithmetic-over-more-than-a-few-numbers","runs":[{"text":"1. Arithmetic over more than a few numbers"}]},{"t":"p","runs":[{"text":"Adding a column of 300 figures is not a language problem, and next-token prediction produces a number that looks right at about the correct magnitude. There is no carry, no column, no check."}]},{"t":"p","runs":[{"text":"The exception matters: several tools now write and run real code to compute, which is different: the model writes a sum and a computer performs it. If your tool shows you the code it ran, the answer is as good as the code. If it just produces a number, do not trust it. Either way, the total still gets checked against your bank balance."}]},{"t":"h3","id":"2-anything-recent","runs":[{"text":"2. Anything recent"}]},{"t":"p","runs":[{"text":"Training stopped on a date. Ask about a rule change from three months ago and you get a fluent account of the old rule, delivered with no hint that time has passed. Web search, where available and switched on, fixes most of this, and you should know whether it is on, because the interface rarely says."}]},{"t":"h3","id":"3-your-private-facts","runs":[{"text":"3. Your private facts"}]},{"t":"p","runs":[{"text":"Your prices, your lead times, your staff rota, what your supplier promised. None of it was in the training data. Attaching your documents fixes this properly; hoping does not."}]},{"t":"h3","id":"4-exact-recall-inside-a-long-document","runs":[{"text":"4. Exact recall inside a long document"}]},{"t":"p","runs":[{"text":"This one surprises people, because it feels like the machine can obviously read. Give it a 90-page contract and ask what clause 14.3 says and it will often be right. Ask it to list every deadline mentioned and it will miss some."}]},{"t":"p","runs":[{"text":"The reason is that attention across a very long input is diluted; material in the middle of a long document is reliably weaker than material at the start or the end. Researchers call it the lost-in-the-middle effect. The practical fix is to work in sections rather than asking one question of the whole thing, and to ask for quotes with page numbers so you can verify what it found."}]},{"t":"h3","id":"5-counting-and-ordering","runs":[{"text":"5. Counting and ordering"}]},{"t":"p","runs":[{"text":"“How many times does this appear?” “Put these 60 items in order.” “Which is the third longest?” These require holding state and iterating, which is not what generating likely text does. Use a spreadsheet. Sorting is a solved problem and has been since the 1950s."}]},{"t":"h3","id":"6-constraint-problems","runs":[{"text":"6. Constraint problems"}]},{"t":"p","runs":[{"text":"A staff rota where Ana cannot work Tuesdays, two people must be on every shift, nobody does more than five days, and Miguel and Sofia cannot be paired. The model produces a rota that looks like a rota and breaks two rules, because it is writing a plausible schedule rather than searching for a valid one."}]},{"t":"p","runs":[{"text":"Rotas, delivery routes and packing are search problems. Real solvers exist and several are free: the scheduling features in booking software, spreadsheet solvers, or open tools like OR-Tools. Ask the model to help you write down the constraints, then give them to something that can actually satisfy them."}]},{"t":"h3","id":"7-anything-that-needs-somebody-accountable","runs":[{"text":"7. Anything that needs somebody accountable"}]},{"t":"p","runs":[{"text":"A filed tax return. A diagnosis. A structural sign-off. Legal advice about your specific situation. These are not hard for technical reasons; they are hard because when they are wrong, a named person answers for it. A model cannot be that person, cannot be insured, and cannot be struck off."}]},{"t":"p","runs":[{"text":"Use it to prepare: to understand the letter, to draft the questions, to organise your papers before an appointment. Never to decide."}]},{"t":"h2","id":"the-pattern-under-all-seven","runs":[{"text":"The pattern under all seven"}]},{"t":"p","runs":[{"text":"Six of these are the same failure wearing different clothes: the machine produces something with the "},{"text":"shape","b":true},{"text":" of a correct answer without any process that could make it correct. A rota that looks like a rota. A total that looks like a total. A clause list that looks complete."}]},{"t":"p","runs":[{"text":"Which gives you a test you can apply to any new task in about five seconds. "},{"text":"Would a wrong answer look different from a right one?","b":true},{"text":" If yes, you are probably fine. If a wrong answer would look identical, you need a source, a calculation or a person, and you need it before the output reaches anybody."}]}],"check":{"question":[{"text":"A restaurant asks a model to build next week’s staff rota from a list of eight constraints. The rota it returns is neatly formatted, covers every shift, and breaks two of the constraints. Why did this happen?"}],"options":[{"text":[{"text":"The constraints were not stated clearly enough, so restating them more precisely would produce a valid rota."}],"correct":false,"tempting":[{"text":"Assumes the failure is comprehension; the model can restate all eight constraints correctly when asked, and still generate a rota that violates them."}]},{"text":[{"text":"Rota-building is a search for an arrangement that satisfies every rule at once, and generating plausible text never performs that search, so a rota-shaped output is not a checked one."}],"correct":true,"tempting":null},{"text":[{"text":"Eight constraints exceed what the model can hold at once, so the earlier ones were forgotten by the time it reached the end of the week and began filling in the remaining shifts."}],"correct":false,"tempting":[{"text":"Blames context length for a reasoning failure; eight short rules occupy a trivial number of tokens and the same failure appears with three."}]},{"text":[{"text":"The model prioritised covering every shift over the individual constraints, which is a reasonable trade-off for it to have made on its own given that an uncovered shift closes the restaurant."}],"correct":false,"tempting":[{"text":"Attributes deliberate prioritisation to a process that does not weigh objectives; nothing in generation evaluates coverage against constraints or trades one for the other."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"the-five-day-tally","title":"The five-day tally, and why memory is not a baseline","minutes":8},"next":{"slug":"one-owner-one-tool","title":"One person owns it, for one month","minutes":8}}}