{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"the-shape-of-a-good-task","title":"The shape of a task it is good at","url":"https://addaly.com/learn/ai-actually-explained/the-shape-of-a-good-task","position":62,"of":92,"minutes":9,"answer":"A model is reliable when the answer is in the material you gave it, cheap to check, and tolerant of occasional error; and unreliable when the answer has to come from its memory.","keyIdea":"A model is reliable when the answer is in the material you gave it, cheap to check, and tolerant of occasional error; and unreliable when the answer has to come from its memory, which is the same mechanism as hallucination seen from the other side."},"body":{"format":"markdown","text":"## Four questions, before you type\n\nEverything in the module on why it goes wrong can be turned around into a test for whether a task is a good one. Four questions do it.\n\n**Is the answer in what I am giving it, or in its memory?** A model rewriting your paragraph, summarising a document you pasted, translating your email, or finding the contradiction between two clauses you supplied is working on material in front of it. A model telling you the population of a district, the date a regulation took effect, or who wrote a paper is working from the compressed residue of its training. The first is where it is strongest; the second is where invention lives. The single biggest improvement most people can make is to move their tasks from the second column to the first.\n\n**Can I check it cheaply?** A rewritten email is checked by reading it. A summary is checked against the source, if you have the source. A piece of code is checked by running it. A citation is checked only by finding the paper, which costs more than the model saved you. If checking costs more than doing, the model has not helped.\n\n**Does it matter if it is wrong one time in twenty?** A first draft, a brainstorm, a name for a variable, a subject line: no. A dosage, a deadline, a contract term, a tax figure: yes. Wrong one time in twenty is roughly what to expect on specific factual detail, and it is fine for the first list and disqualifying for the second.\n\n**Is this common on the web?** The model has dense support for anything millions of people have written about and thin support for anything few have. A Python function to sort a list is dense. A configuration for one obscure device is thin, and it will be answered with equal fluency.\n\n> Figure: Two questions that sort most tasks. A two by two grid, rows Answer is in the material you gave it and Answer must come from its memory, columns Cheap to check and Expensive to check: Answer is in the material you gave it and Cheap to check: Summarise, rewrite, translate, find the contradiction; Answer is in the material you gave it and Expensive to check: A long synthesis you will read closely later; Answer must come from its memory and Cheap to check: A fact you can look up in a minute; Answer must come from its memory and Expensive to check: A citation, a dosage, a deadline, a tax threshold. Moving work from the bottom row to the top row is the single biggest improvement most people can make. The bottom-right box is where every documented disaster in this course lives, because a fluent answer you cannot cheaply check is the one you will not check.\n\n## The frontier is jagged\n\nA 2023 experiment with several hundred consultants at a large firm gave one group a chatbot and set them realistic tasks. On tasks inside what the model could do, the assisted group produced work rated forty per cent higher in quality and finished a quarter faster. On a task deliberately chosen to sit just outside, one that needed a judgement the model's data did not support, the assisted group was nineteen percentage points *less* likely to reach the right answer than the unassisted group. The authors called the boundary a jagged frontier: tasks that look equally hard to a person are not equally hard to the model, and the boundary between the two is invisible from outside.\n\nThat experiment is the honest limit of this lesson's four questions. They tell you which side of the frontier a task is *likely* on. They do not draw the line, and the fluent answer will not tell you which side you are on either. The only reliable way to learn the frontier for your own work is to check the model's output on your own tasks, for a while, and keep a note of where it failed.\n\n## Tasks that pass the four questions\n\nRewriting for tone, length or audience. Summarising a document you have. Turning notes into prose and prose into bullet points. Drafting a reply you will edit. Explaining a concept you can then verify against a textbook. Producing several options to choose among. Translating between major languages, for the gist. Writing code you will run, with tests. Finding what is inconsistent, missing or unclear in a document you wrote. Turning a question you cannot phrase into three questions you can search.\n\n## Tasks that fail them\n\nAnything whose value is a specific fact you cannot check: a citation, a statistic, a date, a price, a phone number, a legal deadline. Arithmetic beyond a few digits, which the module on counting explained; use a calculator, or ask the model to write the calculation for one. Anything about events after its cut-off without a search tool attached. Anything about you, your account, your order, your file, unless the product has actually given it those things. And anything where the fluent, confident, well-structured answer is itself the danger, because you will not check it, which is the reason the last lesson of this module exists.\n\n## The habit\n\nBefore a task, ask the four questions. After it, note whether the answer was right. Within a month you will have a map of the frontier for your own work that no course could give you, and you will have moved most of your use to the side where it is reliable. That is the whole skill, and it is a skill rather than a fact, which is why it cannot simply be told."},"check":{"question":"Which of these tasks is a model most likely to do reliably, according to the mechanism in this lesson?","options":[{"text":"Telling you the correct filing deadline for a particular tax form in your state, which it has seen discussed thousands of times.","correct":false,"tempting":"The answer must come from the model's memory, is a specific detail, and the cost of being wrong is high."},{"text":"Finding the phone number of a small clinic in your town.","correct":false,"tempting":"A rarely written-about specific that the model will produce with the same fluency as a real one."},{"text":"Reading a two-page contract you pasted in and listing the places where it contradicts itself.","correct":true},{"text":"Giving the exact population of your district from the last census.","correct":false,"tempting":"A number recalled from memory, invented at least as often as it is remembered, and neat when invented."}]},"prev":{"slug":"one-model-many-senses","title":"One model, many senses","url":"https://addaly.com/learn/ai-actually-explained/one-model-many-senses"},"next":{"slug":"giving-it-what-it-needs","title":"Give it the material, not the question","url":"https://addaly.com/learn/ai-actually-explained/giving-it-what-it-needs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The shape of a task it is good at', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/the-shape-of-a-good-task (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The shape of a task it is good at","courseTitle":"AI, Actually Explained","module":{"index":7,"slug":"putting-it-to-work","title":"Putting it to work without getting burned"},"answer":null,"keyIdea":"A model is reliable when the answer is in the material you gave it, cheap to check, and tolerant of occasional error; and unreliable when the answer has to come from its memory, which is the same mechanism as hallucination seen from the other side.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 7.1","figure":{"cols":["Cheap to check","Expensive to check"],"rows":["Answer is in the material you gave it","Answer must come from its memory"],"type":"matrix","cells":[["Summarise, rewrite, translate, find the contradiction","A long synthesis you will read closely later"],["A fact you can look up in a minute","A citation, a dosage, a deadline, a tax threshold"]],"title":"Two questions that sort most tasks","caption":"Moving work from the bottom row to the top row is the single biggest improvement most people can make. The bottom-right box is where every documented disaster in this course lives, because a fluent answer you cannot cheaply check is the one you will not check.","cellLabels":[["where it is strongest","usable, with a check built in"],["fine, if you actually look","where invention lives"]]},"alt":"Figure: Two questions that sort most tasks. A two by two grid, rows Answer is in the material you gave it and Answer must come from its memory, columns Cheap to check and Expensive to check: Answer is in the material you gave it and Cheap to check: Summarise, rewrite, translate, find the contradiction; Answer is in the material you gave it and Expensive to check: A long synthesis you will read closely later; Answer must come from its memory and Cheap to check: A fact you can look up in a minute; Answer must come from its memory and Expensive to check: A citation, a dosage, a deadline, a tax threshold. Moving work from the bottom row to the top row is the single biggest improvement most people can make. The bottom-right box is where every documented disaster in this course lives, because a fluent answer you cannot cheaply check is the one you will not check."}},"blocks":[{"t":"h2","id":"four-questions-before-you-type","runs":[{"text":"Four questions, before you type"}]},{"t":"p","runs":[{"text":"Everything in the module on why it goes wrong can be turned around into a test for whether a task is a good one. Four questions do it."}]},{"t":"p","runs":[{"text":"Is the answer in what I am giving it, or in its memory?","b":true},{"text":" A model rewriting your paragraph, summarising a document you pasted, translating your email, or finding the contradiction between two clauses you supplied is working on material in front of it. A model telling you the population of a district, the date a regulation took effect, or who wrote a paper is working from the compressed residue of its training. The first is where it is strongest; the second is where invention lives. The single biggest improvement most people can make is to move their tasks from the second column to the first."}]},{"t":"p","runs":[{"text":"Can I check it cheaply?","b":true},{"text":" A rewritten email is checked by reading it. A summary is checked against the source, if you have the source. A piece of code is checked by running it. A citation is checked only by finding the paper, which costs more than the model saved you. If checking costs more than doing, the model has not helped."}]},{"t":"p","runs":[{"text":"Does it matter if it is wrong one time in twenty?","b":true},{"text":" A first draft, a brainstorm, a name for a variable, a subject line: no. A dosage, a deadline, a contract term, a tax figure: yes. Wrong one time in twenty is roughly what to expect on specific factual detail, and it is fine for the first list and disqualifying for the second."}]},{"t":"p","runs":[{"text":"Is this common on the web?","b":true},{"text":" The model has dense support for anything millions of people have written about and thin support for anything few have. A Python function to sort a list is dense. A configuration for one obscure device is thin, and it will be answered with equal fluency."}]},{"t":"h2","id":"the-frontier-is-jagged","runs":[{"text":"The frontier is jagged"}]},{"t":"p","runs":[{"text":"A 2023 experiment with several hundred consultants at a large firm gave one group a chatbot and set them realistic tasks. On tasks inside what the model could do, the assisted group produced work rated forty per cent higher in quality and finished a quarter faster. On a task deliberately chosen to sit just outside, one that needed a judgement the model’s data did not support, the assisted group was nineteen percentage points "},{"text":"less","i":true},{"text":" likely to reach the right answer than the unassisted group. The authors called the boundary a jagged frontier: tasks that look equally hard to a person are not equally hard to the model, and the boundary between the two is invisible from outside."}]},{"t":"p","runs":[{"text":"That experiment is the honest limit of this lesson’s four questions. They tell you which side of the frontier a task is "},{"text":"likely","i":true},{"text":" on. They do not draw the line, and the fluent answer will not tell you which side you are on either. The only reliable way to learn the frontier for your own work is to check the model’s output on your own tasks, for a while, and keep a note of where it failed."}]},{"t":"h2","id":"tasks-that-pass-the-four-questions","runs":[{"text":"Tasks that pass the four questions"}]},{"t":"p","runs":[{"text":"Rewriting for tone, length or audience. Summarising a document you have. Turning notes into prose and prose into bullet points. Drafting a reply you will edit. Explaining a concept you can then verify against a textbook. Producing several options to choose among. Translating between major languages, for the gist. Writing code you will run, with tests. Finding what is inconsistent, missing or unclear in a document you wrote. Turning a question you cannot phrase into three questions you can search."}]},{"t":"h2","id":"tasks-that-fail-them","runs":[{"text":"Tasks that fail them"}]},{"t":"p","runs":[{"text":"Anything whose value is a specific fact you cannot check: a citation, a statistic, a date, a price, a phone number, a legal deadline. Arithmetic beyond a few digits, which the module on counting explained; use a calculator, or ask the model to write the calculation for one. Anything about events after its cut-off without a search tool attached. Anything about you, your account, your order, your file, unless the product has actually given it those things. And anything where the fluent, confident, well-structured answer is itself the danger, because you will not check it, which is the reason the last lesson of this module exists."}]},{"t":"h2","id":"the-habit","runs":[{"text":"The habit"}]},{"t":"p","runs":[{"text":"Before a task, ask the four questions. After it, note whether the answer was right. Within a month you will have a map of the frontier for your own work that no course could give you, and you will have moved most of your use to the side where it is reliable. That is the whole skill, and it is a skill rather than a fact, which is why it cannot simply be told."}]}],"check":{"question":[{"text":"Which of these tasks is a model most likely to do reliably, according to the mechanism in this lesson?"}],"options":[{"text":[{"text":"Telling you the correct filing deadline for a particular tax form in your state, which it has seen discussed thousands of times."}],"correct":false,"tempting":[{"text":"The answer must come from the model’s memory, is a specific detail, and the cost of being wrong is high."}]},{"text":[{"text":"Finding the phone number of a small clinic in your town."}],"correct":false,"tempting":[{"text":"A rarely written-about specific that the model will produce with the same fluency as a real one."}]},{"text":[{"text":"Reading a two-page contract you pasted in and listing the places where it contradicts itself."}],"correct":true,"tempting":null},{"text":[{"text":"Giving the exact population of your district from the last census."}],"correct":false,"tempting":[{"text":"A number recalled from memory, invented at least as often as it is remembered, and neat when invented."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"one-model-many-senses","title":"One model, many senses","minutes":9},"next":{"slug":"giving-it-what-it-needs","title":"Give it the material, not the question","minutes":8}}}