Why a prompt stops working

It worked in March

A prompt that has been fine for months starts producing worse output. Nothing was changed. This is common, and there are five causes worth checking in order, because four of them are not the prompt.

1. The model changed under you

Providers update models continuously. Sometimes the name changes and you can see it; sometimes the same name points at a new build. Behaviour shifts: verbosity, formatting habits, how strictly instructions are followed, where refusal boundaries sit.

A prompt tuned tightly around one model’s habits is the most fragile kind. If your prompt contains a line that exists to work around something (a repeated instruction because it kept forgetting, a strange phrasing that happened to help) that line is a candidate for the first thing to break.

What to do: pin a model version where the product lets you. Record which model your prompt was tested on. Re-run the ten items after any announced update.

2. The product changed

The system prompt, the tools, the memory behaviour, whether files are retrieved or supplied in full. These change more often than models and are announced less.

A prompt that relied on a web search happening automatically will fail quietly when routing changes and the search stops firing.

3. Your inputs changed shape

The supplier redesigned their invoice. The export added a column. Somebody started sending PDFs where they used to send text. The prompt is unchanged and the thing it operates on is not.

This is the most common cause in practice and the easiest to miss, because you look at the prompt rather than the input.

What to do: when quality drops, look at three recent inputs beside three old ones before you touch anything.

4. The world moved past your context

A prompt with hard-coded facts (this year’s prices, the current policy, a rate, a deadline) becomes confidently wrong on a date nobody marked. It does not fail. It succeeds at applying stale information.

What to do: keep facts out of the prompt where you can, and put them in the pasted material instead. Where a fact must live in the prompt, date it in a comment: # VAT rate as at 2026-04, check annually.

5. You changed it and forgot

Common, and the reason for the change log in an earlier lesson.

The brittleness audit

You can predict fragility before it bites. Read your prompt and mark every line that is:

• A workaround for a behaviour rather than a statement of what you want.
• A hard-coded fact that has a shelf life.
• A reference to product behaviour (“search the web for”, “use the attached file”) that depends on features being enabled.
• A magic phrase with no mechanism you can explain.

Those lines are your breakage surface. Reducing it is straightforward: state what you want rather than how to behave, move facts into the material, and delete anything you cannot justify.

The general principle: a prompt that says what a good answer looks like ages well; a prompt that says how to behave ages badly, because the behaviour it compensates for is exactly what updates change.

Should you upgrade?

Usually yes, and re-test rather than assuming. Newer models generally need less scaffolding, so an upgrade is a good moment to try deleting things: the three examples, the repeated instruction, the elaborate reasoning scaffold. Often two of them are no longer earning their space, and you get a shorter, cheaper, faster prompt out of the exercise.

Occasionally an upgrade is worse for your specific task. That is real, it is not usually admitted, and it is exactly what your ten test items are for.

The version nobody records

One more habit, and it is the cheapest insurance in this module. When you finish tuning a prompt, write down the model and the date beside it.

Six months later, “it used to work” is not a diagnosis anybody can act on. “It scored 9/10 on 4 September against this model, and 5/10 today” is a report you can take to a colleague, a vendor, or your own next hour. The difference between those two sentences is fifteen seconds of typing at a moment when you feel finished and least inclined to type anything.