Diagnosing a bad answer

Whose fault was it

When an answer is unusable, students do one of two things: blame the tool and give up, or blame themselves and rewrite the prompt at random. Neither is a diagnosis. There are five causes, they have different fixes, and you can usually tell them apart in about twenty seconds.

One: the question was underspecified

Symptoms. The answer is generic, starts at the beginning, covers the topic rather than the task, or is a balanced overview when you wanted a position.

Fix. The four elements: level, task, constraints, form. This is the most common cause by a distance, and the most common single omission is level.

Two: the model lacked the material

Symptoms. Vague where you needed specifics. Wrong about your course’s notation, a local regulation, this year’s syllabus, a niche paper, anything after its training cutoff.

Fix. Supply it. Paste the notes, the handbook, the paper. If you cannot supply it because it is not in your possession, this is not a prompting problem and no rewording will help. Go and find the source.

Three: the model had it and got it wrong anyway

Symptoms. A confident specific that turns out to be false. An invented reference. A subtly wrong step in a derivation that otherwise looks fine.

Fix. Verification, not rewording. Ask again cold in a fresh chat; check the specific against a real source; run the computation in a deterministic tool. If it is wrong in the same way every time, the error is well-represented in training data, often a common misconception, and that is worth understanding rather than working around.

Four: the conversation went bad

Symptoms. It contradicts something it said earlier. It reintroduces an approach you rejected. It agrees with everything you say. It has forgotten a constraint you set.

Fix. New chat, carrying a short summary. Not more argument.

Five: the task is not one this tool does

Symptoms. Counting, exact arithmetic, alphabetising, checking whether something exists, telling you what happened last week, remembering your previous conversation.

Fix. A different tool, per Module 1. No prompt fixes a category error.

A worked diagnosis

A student asks for help with a statistics coursework and gets an answer recommending a t-test. The data is ordinal, from a questionnaire, with unequal group sizes.

• Was the question underspecified? Yes: they said compare two groups and never said the outcome was a five-point Likert item. Cause one.
• Did the model lack material? Partly: it never saw the data or the assignment brief. Cause two.
• Was it wrong anyway? Not exactly. Given what it was told, a t-test is a defensible default.

The fix is not a cleverer prompt. It is: state the measurement level, the design, the sample sizes and the actual question, and paste the brief. Then ask which tests are defensible here and what does each assume, and choose. Then run it in jamovi or JASP, which are free.

Notice that the diagnosis also taught the student something: they had not registered that measurement level determines the test. That is usually true. A bad answer often marks the exact place where your own understanding of the problem is thin, which is why diagnosing it beats rewording it.

The cause students most resist naming

Cause five (wrong tool) is the one people argue with. It feels like giving up. It is not; it is the same judgement a lab technician makes when they reach for a balance instead of a ruler. If the job is exact arithmetic, existence-checking, counting, or knowing what happened last Tuesday, no phrasing rescues it. The most experienced users of these tools are conspicuously quick to close the window and open something else, and that speed is most of what makes them look skilled.

The twenty-second routine

When an answer is bad, ask in this order:

1. Did I tell it who I am and what job I want done? (one)
2. Could it possibly have known this? (two, five)
3. Is the wrong part a specific: a name, number, date, reference? (three)
4. How long is this conversation? (four)

One pass through those four questions puts almost every failure in the right box, and each box has one fix. The point is not to become good at prompting. It is to stop wasting time rewording a question that was never the problem.