Epochs, overfitting and checkpoint selection

One to three, and why

The standard advice for supervised fine-tuning is one to three epochs, and unusually for standard advice it is well grounded.

An epoch is one pass over your dataset. On the second pass the model sees examples it has seen before, and a model with billions of parameters and a few thousand examples is entirely capable of memorising them. What it memorises is not the skill you wanted: it is the specific answers, including their specific wording and the specific mistakes in your annotation.

The signature is unmistakable once you have seen it: at each epoch boundary the training loss takes a step down, a sharp discontinuity rather than a smooth continuation. That step is the model recognising the data. It is not learning anything at that moment; it is recalling.

But small datasets need steps

There is a real tension here. With 800 examples at an effective batch of 16, one epoch is 50 optimiser steps. Fifty steps will not move a freshly-initialised adapter anywhere useful.

So the honest framing is not “how many epochs” but “how many optimiser steps do I need, and can I get them without repeating data too often”. Roughly:

• Under 500 steps total, an adapter is usually undertrained.
• 500 to 2,000 steps is the normal range for a task LoRA.
• If reaching 500 steps requires six epochs over 600 examples, the problem is that you have 600 examples. Either collect more, or accept a smaller effective batch to get more steps from the same data: the latter is a reasonable trade at this scale.

LoRA tolerates repetition better than full fine-tuning does, because the low-rank constraint limits how precisely it can memorise. That is a real advantage of the method and not usually mentioned as one.

Checkpoint selection, done properly

Save a checkpoint every few hundred steps. At the end, you have five or ten candidates, and the question is which to ship.

Do not pick on validation loss. As the previous lesson noted, validation loss can rise while your task metric improves, because a more confident model is penalised harder on the items it still gets wrong. If you early-stop on loss you will systematically ship earlier, blander checkpoints.

Pick on your task metric, evaluated on a held-out set, with the canary set alongside it:

for ckpt in checkpoints:
    task   = evaluate(ckpt, held_out)       # what you are trying to improve
    canary = evaluate(ckpt, general_set)    # what must not break
    print(ckpt, round(task, 3), round(canary, 3))

The shape you usually see: task climbs quickly then plateaus, canary is flat then starts falling. The checkpoint to ship is the last one before the canary moves, which is frequently not the last checkpoint and frequently not the best task score either.

Reading the curve honestly

A few readings that are actually diagnostic, beyond the epoch step:

• Train and validation loss both falling, gap steady. Healthy. Keep going.
• Train falling, validation flat. Learning your data’s idiosyncrasies. Not yet harmful, but the marginal epoch is buying little.
• Train falling, validation rising, task metric rising. Not necessarily overfitting in the harmful sense. Check the canary before you panic.
• Train falling, validation rising, task metric flat or falling. Overfitting, straightforwardly. Stop, and go back a checkpoint or two.
• Loss lower on validation than on training. Usually means your validation set is easier, or (much more often) that some validation examples also appear in training. Check for leakage before you celebrate.

The thing that is not in the curve

Catastrophic forgetting does not show up on either loss curve. Both are computed on data from your task distribution, and the capability you have destroyed is somewhere else entirely: the other language your product supports, the ability to say “I do not know”, the refusal that used to work.

This is the argument for running the canary set at every checkpoint rather than at the end. It costs a few minutes per checkpoint, and it is the only instrument that points at the damage while you can still choose a different checkpoint. A run that is evaluated only at the end gives you one artefact and no alternatives.