Change one thing at a time

The rewrite that teaches you nothing

The output is disappointing. You rewrite the prompt: add three examples, tighten the instruction, add a format spec, and paste more context. The new output is better.

What helped? You have no idea, and you have no idea what hurt either: one of those four changes may have made things worse while the other three carried it.

This matters because prompts accumulate. Six months on, that prompt has twelve elements, four of them useless, two of them harmful, and nobody dares remove anything because nobody knows which is which. Every team that uses these tools seriously ends up with a prompt like that.

The discipline

One change. Same test items. Several runs. Keep or discard.

That is the whole method, and each part matters.

One change so the result is attributable.

The same test items: five to ten, with known answers, held out from the prompt. Without them you are comparing today’s impression to yesterday’s memory.

Several runs because of sampling. Two runs of the old version and two of the new is the bare minimum; three each is better. A change that shows up only in one run out of three has not shown up.

Keep or discard explicitly. A change that did not help comes back out. This is the step everybody skips, and it is why prompts get long.

What counts as one change

Adding examples is one change. Adding examples and rewording the task is two.

Rewording the task is one change. Rewording it and adding a format specification is two.

The unit is a hypothesis: I think the failures are caused by X, so I am doing Y. If you cannot state that sentence about a change, you are decorating rather than iterating.

Removal is a change too

Once a prompt is working, run the experiment backwards. Take out the oldest element nobody remembers adding. Run the test. If the score does not move, it stays out.

This is more valuable than it sounds. A shorter prompt is cheaper, faster, easier to read, and less likely to contain a constraint quietly fighting another one. Most inherited prompts can lose a third of their length with no measurable loss, and the exercise usually surfaces one instruction that was actively harmful.

When you cannot be this rigorous

Most of the time you cannot, and should not try. For a one-off task in a chat window, iterate however you like: you will read the answer, so the feedback is immediate and complete.

The discipline is for prompts that will be reused: run over a batch, used weekly, shared with colleagues, or embedded in something. That is where an unattributed improvement becomes a permanent mystery.

The threshold is roughly: will anybody, including me, run this again without reading every output? If yes, do it properly. It costs half an hour once.

Keep the record

Three lines per change, in the same file as the prompt:

2026-09-02  Added 3 boundary examples (refund/delivery edge)
            8/10 → 9/10 over 3 runs. Kept.
2026-09-04  Added "be thorough and precise" to the opening.
            9/10 → 9/10. Removed.
2026-09-11  Required a quoted sentence per extracted value.
            9/10 → 10/10, and the failures became visible. Kept.

That record is worth more than the prompt. It stops you re-trying things that did not work, it tells a colleague why the prompt looks the way it does, and on the day a model update changes everything it tells you what to re-test first.

The order to try things in

When several changes are candidates, try them in order of expected payoff, which is roughly the order of the four causes from the last lesson.

1. Add missing material. Biggest effect, almost always.
2. Sharpen the task: verb, object, finished state.
3. Add a fallback for the case the prompt does not cover. Small change, and it converts invisible failures into visible ones, which changes what you can measure next.
4. Add or change examples. Real effect, real cost in space.
5. Change the format. Usually about usability rather than accuracy.
6. Adjust phrasing. Last, and honestly, mostly not worth the attempt.

Most people start at six and work upwards, which is why iteration feels unrewarding to them. Starting at one is the same amount of typing and a different experience.