{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"did-it-help","title":"Did It Help, and What Did It Break","url":"https://addaly.com/learn/fine-tuning/did-it-help","position":67,"of":84,"minutes":10,"answer":"A win rate without an interval and a canary set is not evidence, it is a feeling.","keyIdea":"A win rate without an interval and a canary set is not evidence, it is a feeling."},"body":{"format":"markdown","text":"## The comparison has to be fair\n\nTwo mistakes make most fine-tuning results meaningless.\n\n**Comparing against a weak baseline.** You spent three weeks on the fine-tune and ten minutes on the prompt you are comparing it against. Run the base model with your *best* prompt (few-shot examples included, retrieval included) on the same held-out inputs. That is the bar. Beating a strawman teaches you nothing.\n\n**Judging your own work unblinded.** Generate both outputs, shuffle which is A and which is B, hide which is which, then judge. If a model does the judging, run each pair twice with the order swapped and discard the pairs where the verdict flips. That flip rate is your judge's error bar, and it is often 10-20%.\n\n## Numbers, with intervals\n\nHead-to-head win rate is the most useful single measure for open-ended tasks. Report it with an interval, because the interval is wider than people expect. For n comparisons, the 95% margin near a 50% rate is roughly `1/sqrt(n)`:\n\n- n = 50 → about ±14 points\n- n = 100 → about ±10 points\n- n = 200 → about ±7 points\n- n = 500 → about ±4 points\n\nSo 58 wins out of 100 is not a result. It is consistent with no difference at all. If your task has an exact answer (a label, an extracted field, valid JSON) use accuracy or exact match instead. Those need fewer samples and start fewer arguments.\n\nPut cost and latency in the same table. A tuned 7B that ties a large hosted model has won, because it costs a fraction to run. A tuned 7B that ties the base 7B has lost, and it is better to know.\n\n## Loss is not your metric\n\nEvaluation loss measures how well the model reproduces text that looks like your training data. It falls while quality falls, routinely. Use it to spot a broken run, not to choose a checkpoint.\n\n## Catastrophic forgetting\n\nTraining pushes weights toward your data, so behaviours you did not train on drift. The narrower your data and the longer you train, the further they drift. LoRA reduces this, because the base weights are untouched, but it does not remove it: the adapter applies to every input, including inputs nothing like your training set.\n\nWhat it looks like in practice:\n\n- Every answer arrives in your training format, including for questions with nothing to do with your task.\n- Answers get shorter and blunter across the board, because your dataset was terse.\n- The model refuses, or answers strangely, outside its new domain.\n- Multi-turn conversation falls apart when you trained only on single turns.\n- **Other languages degrade first.** This is the one teams miss. If the base handled Hindi, Spanish or Arabic and you trained only in English, non-English quality can drop sharply while your English metrics look untouched.\n- Code formatting breaks, or your schema starts appearing inside code blocks.\n\n## The canary set\n\nBuild fifty prompts with nothing to do with your task. General questions, a short reasoning problem, a code snippet, a multi-turn exchange, and, if you serve them, the same questions in each language you support. Run them on the base model and save the outputs. Run them on every fine-tuned checkpoint. Read them side by side yourself. This is not a metric, it is a look.\n\nTen minutes per checkpoint, and it catches the failure that no held-out set built from your own task will ever show you.\n\n## When you find forgetting\n\nIn roughly this order:\n\n1. Fewer epochs. Take the epoch-one checkpoint.\n2. Lower the learning rate: 2e-4, then 1e-4, then 5e-5.\n3. Lower the rank.\n4. Mix 5-20% general instruction data into your training set.\n5. Narrow `target_modules` back toward attention only.\n\nKeep the adapter unmerged while you test, so turning the fine-tune off is one flag rather than a redeploy.\n\n## The decision\n\nShip if the tuned model beats the best-prompt baseline by more than the interval, the canary set is intact, and cost or latency improved or held. Otherwise you still have a result: one that saved you from serving something worse."},"check":{"question":"Your fine-tuned model wins 58 of 100 blind head-to-head comparisons against the base model on your held-out set. The team wants to ship. What is the honest reading?","options":[{"text":"A 58% win rate is a clear majority and the direction is right, so ship it.","correct":false,"tempting":"The direction is genuinely more likely than not, but the 95% interval at n=100 comfortably covers 50%, which means it also covers no difference at all."},{"text":"It failed to clear 60%, so the fine-tune should be abandoned.","correct":false,"tempting":"Setting a bar before you look is good practice, but a threshold narrower than your error bar rejects real improvements as readily as it rejects noise."},{"text":"58 of 100 sits inside a noise band of roughly ten points, so nothing is established yet: you need more comparisons, the strongest-prompt baseline, and a canary check for what broke.","correct":true},{"text":"Re-judge the same 100 pairs with a different judge model to confirm the result.","correct":false,"tempting":"Judge disagreement is real and worth measuring, but re-judging the same items does nothing about sampling error across items, which is what dominates here."}]},"prev":{"slug":"tabular-and-time-series","title":"Tabular and time series: the honest answer","url":"https://addaly.com/learn/fine-tuning/tabular-and-time-series"},"next":{"slug":"the-baseline-you-must-beat","title":"The baseline you must beat","url":"https://addaly.com/learn/fine-tuning/the-baseline-you-must-beat"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Did It Help, and What Did It Break', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/did-it-help (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Did It Help, and What Did It Break","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"A win rate without an interval and a canary set is not evidence, it is a feeling.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-comparison-has-to-be-fair","runs":[{"text":"The comparison has to be fair"}]},{"t":"p","runs":[{"text":"Two mistakes make most fine-tuning results meaningless."}]},{"t":"p","runs":[{"text":"Comparing against a weak baseline.","b":true},{"text":" You spent three weeks on the fine-tune and ten minutes on the prompt you are comparing it against. Run the base model with your "},{"text":"best","i":true},{"text":" prompt (few-shot examples included, retrieval included) on the same held-out inputs. That is the bar. Beating a strawman teaches you nothing."}]},{"t":"p","runs":[{"text":"Judging your own work unblinded.","b":true},{"text":" Generate both outputs, shuffle which is A and which is B, hide which is which, then judge. If a model does the judging, run each pair twice with the order swapped and discard the pairs where the verdict flips. That flip rate is your judge’s error bar, and it is often 10-20%."}]},{"t":"h2","id":"numbers-with-intervals","runs":[{"text":"Numbers, with intervals"}]},{"t":"p","runs":[{"text":"Head-to-head win rate is the most useful single measure for open-ended tasks. Report it with an interval, because the interval is wider than people expect. For n comparisons, the 95% margin near a 50% rate is roughly "},{"text":"1/sqrt(n)","code":true},{"text":":"}]},{"t":"ul","items":[[{"text":"n = 50 → about ±14 points"}],[{"text":"n = 100 → about ±10 points"}],[{"text":"n = 200 → about ±7 points"}],[{"text":"n = 500 → about ±4 points"}]]},{"t":"p","runs":[{"text":"So 58 wins out of 100 is not a result. It is consistent with no difference at all. If your task has an exact answer (a label, an extracted field, valid JSON) use accuracy or exact match instead. Those need fewer samples and start fewer arguments."}]},{"t":"p","runs":[{"text":"Put cost and latency in the same table. A tuned 7B that ties a large hosted model has won, because it costs a fraction to run. A tuned 7B that ties the base 7B has lost, and it is better to know."}]},{"t":"h2","id":"loss-is-not-your-metric","runs":[{"text":"Loss is not your metric"}]},{"t":"p","runs":[{"text":"Evaluation loss measures how well the model reproduces text that looks like your training data. It falls while quality falls, routinely. Use it to spot a broken run, not to choose a checkpoint."}]},{"t":"h2","id":"catastrophic-forgetting","runs":[{"text":"Catastrophic forgetting"}]},{"t":"p","runs":[{"text":"Training pushes weights toward your data, so behaviours you did not train on drift. The narrower your data and the longer you train, the further they drift. LoRA reduces this, because the base weights are untouched, but it does not remove it: the adapter applies to every input, including inputs nothing like your training set."}]},{"t":"p","runs":[{"text":"What it looks like in practice:"}]},{"t":"ul","items":[[{"text":"Every answer arrives in your training format, including for questions with nothing to do with your task."}],[{"text":"Answers get shorter and blunter across the board, because your dataset was terse."}],[{"text":"The model refuses, or answers strangely, outside its new domain."}],[{"text":"Multi-turn conversation falls apart when you trained only on single turns."}],[{"text":"Other languages degrade first.","b":true},{"text":" This is the one teams miss. If the base handled Hindi, Spanish or Arabic and you trained only in English, non-English quality can drop sharply while your English metrics look untouched."}],[{"text":"Code formatting breaks, or your schema starts appearing inside code blocks."}]]},{"t":"h2","id":"the-canary-set","runs":[{"text":"The canary set"}]},{"t":"p","runs":[{"text":"Build fifty prompts with nothing to do with your task. General questions, a short reasoning problem, a code snippet, a multi-turn exchange, and, if you serve them, the same questions in each language you support. Run them on the base model and save the outputs. Run them on every fine-tuned checkpoint. Read them side by side yourself. This is not a metric, it is a look."}]},{"t":"p","runs":[{"text":"Ten minutes per checkpoint, and it catches the failure that no held-out set built from your own task will ever show you."}]},{"t":"h2","id":"when-you-find-forgetting","runs":[{"text":"When you find forgetting"}]},{"t":"p","runs":[{"text":"In roughly this order:"}]},{"t":"ol","start":1,"items":[[{"text":"Fewer epochs. Take the epoch-one checkpoint."}],[{"text":"Lower the learning rate: 2e-4, then 1e-4, then 5e-5."}],[{"text":"Lower the rank."}],[{"text":"Mix 5-20% general instruction data into your training set."}],[{"text":"Narrow "},{"text":"target_modules","code":true},{"text":" back toward attention only."}]]},{"t":"p","runs":[{"text":"Keep the adapter unmerged while you test, so turning the fine-tune off is one flag rather than a redeploy."}]},{"t":"h2","id":"the-decision","runs":[{"text":"The decision"}]},{"t":"p","runs":[{"text":"Ship if the tuned model beats the best-prompt baseline by more than the interval, the canary set is intact, and cost or latency improved or held. Otherwise you still have a result: one that saved you from serving something worse."}]}],"check":{"question":[{"text":"Your fine-tuned model wins 58 of 100 blind head-to-head comparisons against the base model on your held-out set. The team wants to ship. What is the honest reading?"}],"options":[{"text":[{"text":"A 58% win rate is a clear majority and the direction is right, so ship it."}],"correct":false,"tempting":[{"text":"The direction is genuinely more likely than not, but the 95% interval at n=100 comfortably covers 50%, which means it also covers no difference at all."}]},{"text":[{"text":"It failed to clear 60%, so the fine-tune should be abandoned."}],"correct":false,"tempting":[{"text":"Setting a bar before you look is good practice, but a threshold narrower than your error bar rejects real improvements as readily as it rejects noise."}]},{"text":[{"text":"58 of 100 sits inside a noise band of roughly ten points, so nothing is established yet: you need more comparisons, the strongest-prompt baseline, and a canary check for what broke."}],"correct":true,"tempting":null},{"text":[{"text":"Re-judge the same 100 pairs with a different judge model to confirm the result."}],"correct":false,"tempting":[{"text":"Judge disagreement is real and worth measuring, but re-judging the same items does nothing about sampling error across items, which is what dominates here."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"tabular-and-time-series","title":"Tabular and time series: the honest answer","minutes":8},"next":{"slug":"the-baseline-you-must-beat","title":"The baseline you must beat","minutes":9}}}