Did It Help, and What Did It Break

The comparison has to be fair

Two mistakes make most fine-tuning results meaningless.

Comparing against a weak baseline. You spent three weeks on the fine-tune and ten minutes on the prompt you are comparing it against. Run the base model with your *best* prompt — few-shot examples included, retrieval included — on the same held-out inputs. That is the bar. Beating a strawman teaches you nothing.

Judging your own work unblinded. Generate both outputs, shuffle which is A and which is B, hide which is which, then judge. If a model does the judging, run each pair twice with the order swapped and discard the pairs where the verdict flips. That flip rate is your judge's error bar, and it is often 10-20%.

Numbers, with intervals

Head-to-head win rate is the most useful single measure for open-ended tasks. Report it with an interval, because the interval is wider than people expect. For n comparisons, the 95% margin near a 50% rate is roughly 1/sqrt(n):

• n = 50 → about ±14 points
• n = 100 → about ±10 points
• n = 200 → about ±7 points
• n = 500 → about ±4 points

So 58 wins out of 100 is not a result. It is consistent with no difference at all. If your task has an exact answer — a label, an extracted field, valid JSON — use accuracy or exact match instead. Those need fewer samples and start fewer arguments.

Put cost and latency in the same table. A tuned 7B that ties a large hosted model has won, because it costs a fraction to run. A tuned 7B that ties the base 7B has lost, and it is better to know.

Loss is not your metric

Evaluation loss measures how well the model reproduces text that looks like your training data. It falls while quality falls, routinely. Use it to spot a broken run, not to choose a checkpoint.

Catastrophic forgetting

Training pushes weights toward your data, so behaviours you did not train on drift. The narrower your data and the longer you train, the further they drift. LoRA reduces this, because the base weights are untouched, but it does not remove it: the adapter applies to every input, including inputs nothing like your training set.

What it looks like in practice:

• Every answer arrives in your training format, including for questions with nothing to do with your task.
• Answers get shorter and blunter across the board, because your dataset was terse.
• The model refuses, or answers strangely, outside its new domain.
• Multi-turn conversation falls apart when you trained only on single turns.
• Other languages degrade first. This is the one teams miss. If the base handled Hindi, Spanish or Arabic and you trained only in English, non-English quality can drop sharply while your English metrics look untouched.
• Code formatting breaks, or your schema starts appearing inside code blocks.

The canary set

Build fifty prompts with nothing to do with your task. General questions, a short reasoning problem, a code snippet, a multi-turn exchange, and — if you serve them — the same questions in each language you support. Run them on the base model and save the outputs. Run them on every fine-tuned checkpoint. Read them side by side yourself. This is not a metric, it is a look.

Ten minutes per checkpoint, and it catches the failure that no held-out set built from your own task will ever show you.

When you find forgetting

In roughly this order:

1. 1Fewer epochs. Take the epoch-one checkpoint.
2. 2Lower the learning rate: 2e-4, then 1e-4, then 5e-5.
3. 3Lower the rank.
4. 4Mix 5-20% general instruction data into your training set.
5. 5Narrow target_modules back toward attention only.

Keep the adapter unmerged while you test, so turning the fine-tune off is one flag rather than a redeploy.

The decision

Ship if the tuned model beats the best-prompt baseline by more than the interval, the canary set is intact, and cost or latency improved or held. Otherwise you still have a result — one that saved you from serving something worse.