Writing the decision down

A page, before any GPU

Fine-tuning projects fail quietly. Three weeks in, nobody can remember what the original complaint was, the evaluation set has been edited twice to be kinder, and the question “is this better than the prompt we had in March” has no answer because the March prompt was deleted. The fix is one page written before anything runs.

It has six parts, and each one is a commitment you can be held to.

1. The failure, quantified

Not “the model is not good enough”. Something like: on 200 held-out support tickets, the current prompt produces valid JSON 78% of the time and routes to the correct queue 84% of the time. Target: 97% valid, 92% correct.

If you cannot write this sentence, you do not have a project yet. You have a feeling, and the most common outcome of training on a feeling is a model that satisfies the feeling and nothing else.

2. The diagnosis

One line naming which of the six causes you found, and the evidence. Thirty failures sorted: 19 adherence, 6 format, 3 knowledge, 2 capability. The paste test resolved all three knowledge cases, so those go to retrieval.

3. The baseline that must be beaten

Write down the exact artefact you are competing against, and freeze it. Not “the current prompt” but the prompt file at a specific commit, with a specific model and specific sampling parameters. A moving baseline is why so many fine-tunes look like wins.

List at least two:

• The cheap baseline. The best prompt you can write in one more day, including few-shot examples and constrained decoding.
• The cost-matched baseline. If the goal is cost, the honest comparison is the smallest untuned model that a good prompt makes adequate, not the frontier model you are currently paying for.

4. The budget and the kill criteria

Two numbers and one sentence:

Budget: 10 engineer-days and $200 of compute. If after 10 days the tuned model has not beaten the frozen prompt baseline by at least 6 points on routing accuracy with a 95% interval that excludes zero, we stop and ship the prompt.

The kill criterion is the part people leave out, and it is the part that makes the record worth writing. Sunk cost is not a hypothesis. Decide the stopping rule while you are still indifferent.

5. What must not get worse

Name the capabilities you are not trying to change but will be blamed for breaking: general instruction following, refusal behaviour on abusive inputs, the other language your product supports, answers in the domain you did not train on. These become the canary set, and it is written before training precisely so that you cannot choose it afterwards to flatter the result.

6. Ownership and the rebuild test

Who owns this artefact, where the dataset lives, and the answer to one question: if the base model is superseded next quarter, how many days to rebuild? If the answer is more than two, fix the pipeline before you train, not after.

The template

# Fine-tune decision: ticket routing, 2026-09

Failure      : 84% routing accuracy, 78% valid JSON on eval-v3 (n=200)
Target       : 92% / 97%
Diagnosis    : 19/30 adherence, 6 format, 3 knowledge (→ retrieval), 2 capability
Baseline     : prompts/route.md @ a91f2c, model X, temp 0
Cost baseline: smallest model that passes with a good prompt
Budget       : 10 days, $200. Kill if < +6 pts with CI excluding 0.
Must not break: refusals, Hindi inputs, the escalation path
Owner        : ___   Dataset: git@…/routing-data   Rebuild: 1 day

Ten lines. It costs twenty minutes and it is the only artefact in the project that is still useful if the fine-tune is abandoned, because a well-written record of why we decided not to is worth as much to the next person as a model would have been.

The uncomfortable part

Write the record honestly and a meaningful fraction of proposed fine-tunes will not survive it. That is the point. The most valuable outcome of this module is the projects it stops, and the second most valuable is that the ones which proceed do so with a frozen baseline, a stated interval and a stopping rule, which is to say, with the conditions under which a result could be believed.