{"course":{"slug":"ai-for-students","title":"AI for Students","url":"https://addaly.com/learn/ai-for-students","level":"beginner","track":"work"},"lesson":{"slug":"diagnosing-a-bad-answer","title":"Diagnosing a bad answer","url":"https://addaly.com/learn/ai-for-students/diagnosing-a-bad-answer","position":17,"of":73,"minutes":8,"answer":"Bad answers have five distinguishable causes (underspecified question, missing material, genuine error, degraded conversation, wrong tool) and each has its own fix, so diagnose before you reword.","keyIdea":"Bad answers have five distinguishable causes (underspecified question, missing material, genuine error, degraded conversation, wrong tool) and each has its own fix, so diagnose before you reword."},"body":{"format":"markdown","text":"## Whose fault was it\n\nWhen an answer is unusable, students do one of two things: blame the tool and give up, or blame themselves and rewrite the prompt at random. Neither is a diagnosis. There are five causes, they have different fixes, and you can usually tell them apart in about twenty seconds.\n\n## One: the question was underspecified\n\n**Symptoms.** The answer is generic, starts at the beginning, covers the topic rather than the task, or is a balanced overview when you wanted a position.\n\n**Fix.** The four elements: level, task, constraints, form. This is the most common cause by a distance, and the most common single omission is level.\n\n## Two: the model lacked the material\n\n**Symptoms.** Vague where you needed specifics. Wrong about your course's notation, a local regulation, this year's syllabus, a niche paper, anything after its training cutoff.\n\n**Fix.** Supply it. Paste the notes, the handbook, the paper. If you cannot supply it because it is not in your possession, this is not a prompting problem and no rewording will help. Go and find the source.\n\n## Three: the model had it and got it wrong anyway\n\n**Symptoms.** A confident specific that turns out to be false. An invented reference. A subtly wrong step in a derivation that otherwise looks fine.\n\n**Fix.** Verification, not rewording. Ask again cold in a fresh chat; check the specific against a real source; run the computation in a deterministic tool. If it is wrong in the same way every time, the error is well-represented in training data, often a common misconception, and that is worth understanding rather than working around.\n\n## Four: the conversation went bad\n\n**Symptoms.** It contradicts something it said earlier. It reintroduces an approach you rejected. It agrees with everything you say. It has forgotten a constraint you set.\n\n**Fix.** New chat, carrying a short summary. Not more argument.\n\n## Five: the task is not one this tool does\n\n**Symptoms.** Counting, exact arithmetic, alphabetising, checking whether something exists, telling you what happened last week, remembering your previous conversation.\n\n**Fix.** A different tool, per Module 1. No prompt fixes a category error.\n\n## A worked diagnosis\n\nA student asks for help with a statistics coursework and gets an answer recommending a t-test. The data is ordinal, from a questionnaire, with unequal group sizes.\n\n- Was the question underspecified? Yes: they said *compare two groups* and never said the outcome was a five-point Likert item. **Cause one.**\n- Did the model lack material? Partly: it never saw the data or the assignment brief. **Cause two.**\n- Was it wrong anyway? Not exactly. Given what it was told, a t-test is a defensible default.\n\nThe fix is not a cleverer prompt. It is: state the measurement level, the design, the sample sizes and the actual question, and paste the brief. Then ask *which tests are defensible here and what does each assume*, and choose. Then run it in jamovi or JASP, which are free.\n\nNotice that the diagnosis also taught the student something: they had not registered that measurement level determines the test. That is usually true. A bad answer often marks the exact place where your own understanding of the problem is thin, which is why diagnosing it beats rewording it.\n\n## The cause students most resist naming\n\nCause five (wrong tool) is the one people argue with. It feels like giving up. It is not; it is the same judgement a lab technician makes when they reach for a balance instead of a ruler. If the job is exact arithmetic, existence-checking, counting, or knowing what happened last Tuesday, no phrasing rescues it. The most experienced users of these tools are conspicuously quick to close the window and open something else, and that speed is most of what makes them look skilled.\n\n## The twenty-second routine\n\nWhen an answer is bad, ask in this order:\n\n1. Did I tell it who I am and what job I want done? *(one)*\n2. Could it possibly have known this? *(two, five)*\n3. Is the wrong part a specific: a name, number, date, reference? *(three)*\n4. How long is this conversation? *(four)*\n\nOne pass through those four questions puts almost every failure in the right box, and each box has one fix. The point is not to become good at prompting. It is to stop wasting time rewording a question that was never the problem."},"check":{"question":"A student's prompt is well specified and the source is pasted in, but the model gives the same wrong definition in three separate fresh conversations. What does the repetition indicate?","options":[{"text":"That the prompt still needs refining, since a consistent failure across attempts points to a consistent flaw in the instruction.","correct":false,"tempting":"Assumes stability implicates the prompt; identical instructions producing identical output is expected, and says nothing about whether the instruction was at fault."},{"text":"That the pasted source was too short for the model to ground on, so it fell back on memory each time.","correct":false,"tempting":"Reaches for a grounding failure when the diagnosis routine has already confirmed the material was supplied and specified."},{"text":"That the model is retrieving a cached answer, so a different provider's model would give the correct version.","correct":false,"tempting":"Invents caching between separate conversations; models generate afresh each time, and a widespread misconception will appear across providers because they share sources."},{"text":"That the error is well represented in the training data, most likely a common misconception worth understanding.","correct":true}]},"prev":{"slug":"what-not-to-paste","title":"What not to paste","url":"https://addaly.com/learn/ai-for-students/what-not-to-paste"},"next":{"slug":"prompts-worth-keeping","title":"Prompts worth keeping","url":"https://addaly.com/learn/ai-for-students/prompts-worth-keeping"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Diagnosing a bad answer', in AI for Students. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-students/diagnosing-a-bad-answer (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Diagnosing a bad answer","courseTitle":"AI for Students","module":{"index":2,"slug":"asking-well","title":"Asking well"},"answer":null,"keyIdea":"Bad answers have five distinguishable causes (underspecified question, missing material, genuine error, degraded conversation, wrong tool) and each has its own fix, so diagnose before you reword.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"whose-fault-was-it","runs":[{"text":"Whose fault was it"}]},{"t":"p","runs":[{"text":"When an answer is unusable, students do one of two things: blame the tool and give up, or blame themselves and rewrite the prompt at random. Neither is a diagnosis. There are five causes, they have different fixes, and you can usually tell them apart in about twenty seconds."}]},{"t":"h2","id":"one-the-question-was-underspecified","runs":[{"text":"One: the question was underspecified"}]},{"t":"p","runs":[{"text":"Symptoms.","b":true},{"text":" The answer is generic, starts at the beginning, covers the topic rather than the task, or is a balanced overview when you wanted a position."}]},{"t":"p","runs":[{"text":"Fix.","b":true},{"text":" The four elements: level, task, constraints, form. This is the most common cause by a distance, and the most common single omission is level."}]},{"t":"h2","id":"two-the-model-lacked-the-material","runs":[{"text":"Two: the model lacked the material"}]},{"t":"p","runs":[{"text":"Symptoms.","b":true},{"text":" Vague where you needed specifics. Wrong about your course’s notation, a local regulation, this year’s syllabus, a niche paper, anything after its training cutoff."}]},{"t":"p","runs":[{"text":"Fix.","b":true},{"text":" Supply it. Paste the notes, the handbook, the paper. If you cannot supply it because it is not in your possession, this is not a prompting problem and no rewording will help. Go and find the source."}]},{"t":"h2","id":"three-the-model-had-it-and-got-it-wrong-anyway","runs":[{"text":"Three: the model had it and got it wrong anyway"}]},{"t":"p","runs":[{"text":"Symptoms.","b":true},{"text":" A confident specific that turns out to be false. An invented reference. A subtly wrong step in a derivation that otherwise looks fine."}]},{"t":"p","runs":[{"text":"Fix.","b":true},{"text":" Verification, not rewording. Ask again cold in a fresh chat; check the specific against a real source; run the computation in a deterministic tool. If it is wrong in the same way every time, the error is well-represented in training data, often a common misconception, and that is worth understanding rather than working around."}]},{"t":"h2","id":"four-the-conversation-went-bad","runs":[{"text":"Four: the conversation went bad"}]},{"t":"p","runs":[{"text":"Symptoms.","b":true},{"text":" It contradicts something it said earlier. It reintroduces an approach you rejected. It agrees with everything you say. It has forgotten a constraint you set."}]},{"t":"p","runs":[{"text":"Fix.","b":true},{"text":" New chat, carrying a short summary. Not more argument."}]},{"t":"h2","id":"five-the-task-is-not-one-this-tool-does","runs":[{"text":"Five: the task is not one this tool does"}]},{"t":"p","runs":[{"text":"Symptoms.","b":true},{"text":" Counting, exact arithmetic, alphabetising, checking whether something exists, telling you what happened last week, remembering your previous conversation."}]},{"t":"p","runs":[{"text":"Fix.","b":true},{"text":" A different tool, per Module 1. No prompt fixes a category error."}]},{"t":"h2","id":"a-worked-diagnosis","runs":[{"text":"A worked diagnosis"}]},{"t":"p","runs":[{"text":"A student asks for help with a statistics coursework and gets an answer recommending a t-test. The data is ordinal, from a questionnaire, with unequal group sizes."}]},{"t":"ul","items":[[{"text":"Was the question underspecified? Yes: they said "},{"text":"compare two groups","i":true},{"text":" and never said the outcome was a five-point Likert item. "},{"text":"Cause one.","b":true}],[{"text":"Did the model lack material? Partly: it never saw the data or the assignment brief. "},{"text":"Cause two.","b":true}],[{"text":"Was it wrong anyway? Not exactly. Given what it was told, a t-test is a defensible default."}]]},{"t":"p","runs":[{"text":"The fix is not a cleverer prompt. It is: state the measurement level, the design, the sample sizes and the actual question, and paste the brief. Then ask "},{"text":"which tests are defensible here and what does each assume","i":true},{"text":", and choose. Then run it in jamovi or JASP, which are free."}]},{"t":"p","runs":[{"text":"Notice that the diagnosis also taught the student something: they had not registered that measurement level determines the test. That is usually true. A bad answer often marks the exact place where your own understanding of the problem is thin, which is why diagnosing it beats rewording it."}]},{"t":"h2","id":"the-cause-students-most-resist-naming","runs":[{"text":"The cause students most resist naming"}]},{"t":"p","runs":[{"text":"Cause five (wrong tool) is the one people argue with. It feels like giving up. It is not; it is the same judgement a lab technician makes when they reach for a balance instead of a ruler. If the job is exact arithmetic, existence-checking, counting, or knowing what happened last Tuesday, no phrasing rescues it. The most experienced users of these tools are conspicuously quick to close the window and open something else, and that speed is most of what makes them look skilled."}]},{"t":"h2","id":"the-twenty-second-routine","runs":[{"text":"The twenty-second routine"}]},{"t":"p","runs":[{"text":"When an answer is bad, ask in this order:"}]},{"t":"ol","start":1,"items":[[{"text":"Did I tell it who I am and what job I want done? "},{"text":"(one)","i":true}],[{"text":"Could it possibly have known this? "},{"text":"(two, five)","i":true}],[{"text":"Is the wrong part a specific: a name, number, date, reference? "},{"text":"(three)","i":true}],[{"text":"How long is this conversation? "},{"text":"(four)","i":true}]]},{"t":"p","runs":[{"text":"One pass through those four questions puts almost every failure in the right box, and each box has one fix. The point is not to become good at prompting. It is to stop wasting time rewording a question that was never the problem."}]}],"check":{"question":[{"text":"A student’s prompt is well specified and the source is pasted in, but the model gives the same wrong definition in three separate fresh conversations. What does the repetition indicate?"}],"options":[{"text":[{"text":"That the prompt still needs refining, since a consistent failure across attempts points to a consistent flaw in the instruction."}],"correct":false,"tempting":[{"text":"Assumes stability implicates the prompt; identical instructions producing identical output is expected, and says nothing about whether the instruction was at fault."}]},{"text":[{"text":"That the pasted source was too short for the model to ground on, so it fell back on memory each time."}],"correct":false,"tempting":[{"text":"Reaches for a grounding failure when the diagnosis routine has already confirmed the material was supplied and specified."}]},{"text":[{"text":"That the model is retrieving a cached answer, so a different provider’s model would give the correct version."}],"correct":false,"tempting":[{"text":"Invents caching between separate conversations; models generate afresh each time, and a widespread misconception will appear across providers because they share sources."}]},{"text":[{"text":"That the error is well represented in the training data, most likely a common misconception worth understanding."}],"correct":true,"tempting":null}],"order":[1,3,2,0],"why":null},"prev":{"slug":"what-not-to-paste","title":"What not to paste","minutes":9},"next":{"slug":"prompts-worth-keeping","title":"Prompts worth keeping","minutes":7}}}