{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"ablations-that-earn-their-cost","title":"Ablations that earn their cost","url":"https://addaly.com/learn/fine-tuning/ablations-that-earn-their-cost","position":75,"of":84,"minutes":8,"answer":"Run an ablation only when its result would change a stated decision.","keyIdea":"Run an ablation only when its result would change a stated decision, and read every result against a three-seed noise floor: the data learning curve is the one that settles whether another week of labelling is worth it, and a written log of non-findings prevents the same experiment being run twice."},"body":{"format":"markdown","text":"## The test for whether to run one\n\nAn ablation is worth running if, and only if, its result would change a decision. Write the decision down before you launch it:\n\n> If more data helps, we spend next week labelling. If it does not, we ship and move on.\n\nThat sentence turns a curiosity into an experiment. Without it, you will run six configurations, produce six numbers, and choose the largest, which is a way of selecting noise.\n\n## The four that usually earn it\n\n**1. The data learning curve.** Train on 25%, 50%, 75% and 100% of your examples. Plot the task metric against the count.\n\nThis is the most valuable ablation in applied fine-tuning, because it answers the question teams argue about most: whether to collect more data. A curve still climbing steeply at 100% says yes. A flat curve from 50% onwards says the next thousand examples will do nothing, and your effort belongs somewhere else: better labels, a different base model, a different approach entirely.\n\n**2. The replay ratio.** 100% task data against 80/20 against 60/40, scored on both the task metric and the canary set. This one usually changes what you ship, because the canary recovery at 80/20 typically costs almost nothing on the task.\n\n**3. Epochs.** You get this free from your checkpoints; no extra runs needed. Evaluate the checkpoint at the end of each epoch on both metrics.\n\n**4. Rank, once.** r = 8 against r = 32 at fixed alpha-to-rank ratio. If they are within noise, and they usually are, you have learned that rank is not your constraint and you can stop thinking about it for this project.\n\n## What it costs\n\nSix short runs on an 8B QLoRA over a few thousand examples is roughly six to ten GPU-hours. On a rented consumer card that is under ten pounds, and on a free tier it is two long sessions.\n\nAgainst that: a week of labelling that turns out to have been unnecessary, or a shipped model that is two points worse than one of the configurations you never tried.\n\n## One factor at a time\n\nChange one thing between runs and hold everything else, including the seed, fixed. It is tempting to change two, since the runs are cheap, but you then cannot attribute the difference and you have spent the compute for nothing.\n\nThe exception: when you believe two factors interact, run the full grid on a small scale rather than two single-factor comparisons. Rank and dataset size plausibly interact; learning rate and epochs certainly do. Four runs instead of two, and an answer you can interpret.\n\n## Against your noise floor\n\nThis is what makes the whole exercise honest. From the reproducibility lesson: run your baseline configuration three times with different seeds and record the spread.\n\n```\nnoise floor (3 seeds):  0.858 – 0.884,  sd ≈ 1.3 pts\n\ndata 25%    0.801\ndata 50%    0.847\ndata 75%    0.869\ndata 100%   0.871     ← flat from 75%; more data will not help\nreplay 80/20 0.870  canary 0.81  (vs canary 0.62 at 100% task)  ← ship this\nrank 32      0.876     ← +0.5 pt, inside noise, ignore\n```\n\nTwo real findings and one non-finding, and the non-finding is as useful as the others because it stops an argument.\n\n## Write it down where it will be found\n\nA file in the repository, one line per run: what changed, the metric, the canary, the date, the checkpoint path.\n\n```\n2026-09-14  baseline s0/s1/s2       .871/.858/.884  canary .62\n2026-09-14  data 50%                .847            canary .64\n2026-09-15  replay 80/20            .870            canary .81   ← shipped\n2026-09-15  rank 32                 .876            canary .60\n```\n\nThree months later somebody will propose raising the rank. This file answers them in ten seconds, and without it the answer is another two days of compute and argument. The log is the durable output of an ablation; the model is the perishable one."},"check":{"question":"An ablation shows task metric 0.847 at 50% of the data, 0.869 at 75% and 0.871 at 100%, against a three-seed noise floor with a standard deviation of 1.3 points. What does this support?","options":[{"text":"Collecting more data is the best investment, since the metric rose across every step of the curve.","correct":false,"tempting":"Reads the whole curve as a trend and ignores that the final step is well inside the noise floor."},{"text":"The dataset should be reduced to 75%, since the last quarter contributes nothing and shorter runs are cheaper.","correct":false,"tempting":"Acts on the flat segment in the wrong direction; removing data that is not harming anything buys a little compute at a real risk."},{"text":"The noise floor is too large to interpret the curve at all, and every point needs three seeds before anything can be concluded.","correct":false,"tempting":"Demands uniform rigour where the early steps clearly exceed the noise; the 4.6-point rise from 50% to 75% is interpretable as it stands."},{"text":"The curve has flattened between 75% and 100%, so further labelling is unlikely to help and effort belongs elsewhere.","correct":true}]},"prev":{"slug":"calibration-and-overconfidence","title":"Calibration after tuning","url":"https://addaly.com/learn/fine-tuning/calibration-and-overconfidence"},"next":{"slug":"serving-the-adapter","title":"Serving the Adapter","url":"https://addaly.com/learn/fine-tuning/serving-the-adapter"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ablations that earn their cost', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/ablations-that-earn-their-cost (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ablations that earn their cost","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"Run an ablation only when its result would change a stated decision, and read every result against a three-seed noise floor: the data learning curve is the one that settles whether another week of labelling is worth it, and a written log of non-findings prevents the same experiment being run twice.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-test-for-whether-to-run-one","runs":[{"text":"The test for whether to run one"}]},{"t":"p","runs":[{"text":"An ablation is worth running if, and only if, its result would change a decision. Write the decision down before you launch it:"}]},{"t":"quote","runs":[{"text":"If more data helps, we spend next week labelling. If it does not, we ship and move on."}]},{"t":"p","runs":[{"text":"That sentence turns a curiosity into an experiment. Without it, you will run six configurations, produce six numbers, and choose the largest, which is a way of selecting noise."}]},{"t":"h2","id":"the-four-that-usually-earn-it","runs":[{"text":"The four that usually earn it"}]},{"t":"p","runs":[{"text":"1. The data learning curve.","b":true},{"text":" Train on 25%, 50%, 75% and 100% of your examples. Plot the task metric against the count."}]},{"t":"p","runs":[{"text":"This is the most valuable ablation in applied fine-tuning, because it answers the question teams argue about most: whether to collect more data. A curve still climbing steeply at 100% says yes. A flat curve from 50% onwards says the next thousand examples will do nothing, and your effort belongs somewhere else: better labels, a different base model, a different approach entirely."}]},{"t":"p","runs":[{"text":"2. The replay ratio.","b":true},{"text":" 100% task data against 80/20 against 60/40, scored on both the task metric and the canary set. This one usually changes what you ship, because the canary recovery at 80/20 typically costs almost nothing on the task."}]},{"t":"p","runs":[{"text":"3. Epochs.","b":true},{"text":" You get this free from your checkpoints; no extra runs needed. Evaluate the checkpoint at the end of each epoch on both metrics."}]},{"t":"p","runs":[{"text":"4. Rank, once.","b":true},{"text":" r = 8 against r = 32 at fixed alpha-to-rank ratio. If they are within noise, and they usually are, you have learned that rank is not your constraint and you can stop thinking about it for this project."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"Six short runs on an 8B QLoRA over a few thousand examples is roughly six to ten GPU-hours. On a rented consumer card that is under ten pounds, and on a free tier it is two long sessions."}]},{"t":"p","runs":[{"text":"Against that: a week of labelling that turns out to have been unnecessary, or a shipped model that is two points worse than one of the configurations you never tried."}]},{"t":"h2","id":"one-factor-at-a-time","runs":[{"text":"One factor at a time"}]},{"t":"p","runs":[{"text":"Change one thing between runs and hold everything else, including the seed, fixed. It is tempting to change two, since the runs are cheap, but you then cannot attribute the difference and you have spent the compute for nothing."}]},{"t":"p","runs":[{"text":"The exception: when you believe two factors interact, run the full grid on a small scale rather than two single-factor comparisons. Rank and dataset size plausibly interact; learning rate and epochs certainly do. Four runs instead of two, and an answer you can interpret."}]},{"t":"h2","id":"against-your-noise-floor","runs":[{"text":"Against your noise floor"}]},{"t":"p","runs":[{"text":"This is what makes the whole exercise honest. From the reproducibility lesson: run your baseline configuration three times with different seeds and record the spread."}]},{"t":"code","lang":"","label":"Text","text":"noise floor (3 seeds):  0.858 – 0.884,  sd ≈ 1.3 pts\n\ndata 25%    0.801\ndata 50%    0.847\ndata 75%    0.869\ndata 100%   0.871     ← flat from 75%; more data will not help\nreplay 80/20 0.870  canary 0.81  (vs canary 0.62 at 100% task)  ← ship this\nrank 32      0.876     ← +0.5 pt, inside noise, ignore"},{"t":"p","runs":[{"text":"Two real findings and one non-finding, and the non-finding is as useful as the others because it stops an argument."}]},{"t":"h2","id":"write-it-down-where-it-will-be-found","runs":[{"text":"Write it down where it will be found"}]},{"t":"p","runs":[{"text":"A file in the repository, one line per run: what changed, the metric, the canary, the date, the checkpoint path."}]},{"t":"code","lang":"","label":"Text","text":"2026-09-14  baseline s0/s1/s2       .871/.858/.884  canary .62\n2026-09-14  data 50%                .847            canary .64\n2026-09-15  replay 80/20            .870            canary .81   ← shipped\n2026-09-15  rank 32                 .876            canary .60"},{"t":"p","runs":[{"text":"Three months later somebody will propose raising the rank. This file answers them in ten seconds, and without it the answer is another two days of compute and argument. The log is the durable output of an ablation; the model is the perishable one."}]}],"check":{"question":[{"text":"An ablation shows task metric 0.847 at 50% of the data, 0.869 at 75% and 0.871 at 100%, against a three-seed noise floor with a standard deviation of 1.3 points. What does this support?"}],"options":[{"text":[{"text":"Collecting more data is the best investment, since the metric rose across every step of the curve."}],"correct":false,"tempting":[{"text":"Reads the whole curve as a trend and ignores that the final step is well inside the noise floor."}]},{"text":[{"text":"The dataset should be reduced to 75%, since the last quarter contributes nothing and shorter runs are cheaper."}],"correct":false,"tempting":[{"text":"Acts on the flat segment in the wrong direction; removing data that is not harming anything buys a little compute at a real risk."}]},{"text":[{"text":"The noise floor is too large to interpret the curve at all, and every point needs three seeds before anything can be concluded."}],"correct":false,"tempting":[{"text":"Demands uniform rigour where the early steps clearly exceed the noise; the 4.6-point rise from 50% to 75% is interpretable as it stands."}]},{"text":[{"text":"The curve has flattened between 75% and 100%, so further labelling is unlikely to help and effort belongs elsewhere."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"calibration-and-overconfidence","title":"Calibration after tuning","minutes":8},"next":{"slug":"serving-the-adapter","title":"Serving the Adapter","minutes":9}}}