Ablations that earn their cost

The test for whether to run one

An ablation is worth running if, and only if, its result would change a decision. Write the decision down before you launch it:

If more data helps, we spend next week labelling. If it does not, we ship and move on.

That sentence turns a curiosity into an experiment. Without it, you will run six configurations, produce six numbers, and choose the largest, which is a way of selecting noise.

The four that usually earn it

1. The data learning curve. Train on 25%, 50%, 75% and 100% of your examples. Plot the task metric against the count.

This is the most valuable ablation in applied fine-tuning, because it answers the question teams argue about most: whether to collect more data. A curve still climbing steeply at 100% says yes. A flat curve from 50% onwards says the next thousand examples will do nothing, and your effort belongs somewhere else: better labels, a different base model, a different approach entirely.

2. The replay ratio. 100% task data against 80/20 against 60/40, scored on both the task metric and the canary set. This one usually changes what you ship, because the canary recovery at 80/20 typically costs almost nothing on the task.

3. Epochs. You get this free from your checkpoints; no extra runs needed. Evaluate the checkpoint at the end of each epoch on both metrics.

4. Rank, once. r = 8 against r = 32 at fixed alpha-to-rank ratio. If they are within noise, and they usually are, you have learned that rank is not your constraint and you can stop thinking about it for this project.

What it costs

Six short runs on an 8B QLoRA over a few thousand examples is roughly six to ten GPU-hours. On a rented consumer card that is under ten pounds, and on a free tier it is two long sessions.

Against that: a week of labelling that turns out to have been unnecessary, or a shipped model that is two points worse than one of the configurations you never tried.

One factor at a time

Change one thing between runs and hold everything else, including the seed, fixed. It is tempting to change two, since the runs are cheap, but you then cannot attribute the difference and you have spent the compute for nothing.

The exception: when you believe two factors interact, run the full grid on a small scale rather than two single-factor comparisons. Rank and dataset size plausibly interact; learning rate and epochs certainly do. Four runs instead of two, and an answer you can interpret.

Against your noise floor

This is what makes the whole exercise honest. From the reproducibility lesson: run your baseline configuration three times with different seeds and record the spread.

noise floor (3 seeds):  0.858 – 0.884,  sd ≈ 1.3 pts

data 25%    0.801
data 50%    0.847
data 75%    0.869
data 100%   0.871     ← flat from 75%; more data will not help
replay 80/20 0.870  canary 0.81  (vs canary 0.62 at 100% task)  ← ship this
rank 32      0.876     ← +0.5 pt, inside noise, ignore

Two real findings and one non-finding, and the non-finding is as useful as the others because it stops an argument.

Write it down where it will be found

A file in the repository, one line per run: what changed, the metric, the canary, the date, the checkpoint path.

2026-09-14  baseline s0/s1/s2       .871/.858/.884  canary .62
2026-09-14  data 50%                .847            canary .64
2026-09-15  replay 80/20            .870            canary .81   ← shipped
2026-09-15  rank 32                 .876            canary .60

Three months later somebody will propose raising the rank. This file answers them in ten seconds, and without it the answer is another two days of compute and argument. The log is the durable output of an ablation; the model is the perishable one.