{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"epochs-and-the-loss-curve","title":"Epochs, overfitting and checkpoint selection","url":"https://addaly.com/learn/fine-tuning/epochs-and-the-loss-curve","position":17,"of":84,"minutes":9,"answer":"The step down in training loss at each epoch boundary is memorisation, not learning; select checkpoints on a task metric with a canary set beside it rather than on validation loss.","keyIdea":"The step down in training loss at each epoch boundary is memorisation, not learning; select checkpoints on a task metric with a canary set beside it rather than on validation loss, because the best checkpoint is often the last one before general capability starts to slip."},"body":{"format":"markdown","text":"## One to three, and why\n\nThe standard advice for supervised fine-tuning is one to three epochs, and unusually for standard advice it is well grounded.\n\nAn epoch is one pass over your dataset. On the second pass the model sees examples it has seen before, and a model with billions of parameters and a few thousand examples is entirely capable of memorising them. What it memorises is not the skill you wanted: it is the specific answers, including their specific wording and the specific mistakes in your annotation.\n\nThe signature is unmistakable once you have seen it: at each epoch boundary the training loss takes a **step down**, a sharp discontinuity rather than a smooth continuation. That step is the model recognising the data. It is not learning anything at that moment; it is recalling.\n\n## But small datasets need steps\n\nThere is a real tension here. With 800 examples at an effective batch of 16, one epoch is 50 optimiser steps. Fifty steps will not move a freshly-initialised adapter anywhere useful.\n\nSo the honest framing is not \"how many epochs\" but \"how many optimiser steps do I need, and can I get them without repeating data too often\". Roughly:\n\n- Under 500 steps total, an adapter is usually undertrained.\n- 500 to 2,000 steps is the normal range for a task LoRA.\n- If reaching 500 steps requires six epochs over 600 examples, the problem is that you have 600 examples. Either collect more, or accept a smaller effective batch to get more steps from the same data: the latter is a reasonable trade at this scale.\n\nLoRA tolerates repetition better than full fine-tuning does, because the low-rank constraint limits how precisely it can memorise. That is a real advantage of the method and not usually mentioned as one.\n\n## Checkpoint selection, done properly\n\nSave a checkpoint every few hundred steps. At the end, you have five or ten candidates, and the question is which to ship.\n\nDo **not** pick on validation loss. As the previous lesson noted, validation loss can rise while your task metric improves, because a more confident model is penalised harder on the items it still gets wrong. If you early-stop on loss you will systematically ship earlier, blander checkpoints.\n\nPick on your task metric, evaluated on a held-out set, with the canary set alongside it:\n\n```python\nfor ckpt in checkpoints:\n    task   = evaluate(ckpt, held_out)       # what you are trying to improve\n    canary = evaluate(ckpt, general_set)    # what must not break\n    print(ckpt, round(task, 3), round(canary, 3))\n```\n\nThe shape you usually see: task climbs quickly then plateaus, canary is flat then starts falling. The checkpoint to ship is the last one before the canary moves, which is frequently *not* the last checkpoint and frequently not the best task score either.\n\n## Reading the curve honestly\n\nA few readings that are actually diagnostic, beyond the epoch step:\n\n- **Train and validation loss both falling, gap steady.** Healthy. Keep going.\n- **Train falling, validation flat.** Learning your data's idiosyncrasies. Not yet harmful, but the marginal epoch is buying little.\n- **Train falling, validation rising, task metric rising.** Not necessarily overfitting in the harmful sense. Check the canary before you panic.\n- **Train falling, validation rising, task metric flat or falling.** Overfitting, straightforwardly. Stop, and go back a checkpoint or two.\n- **Loss lower on validation than on training.** Usually means your validation set is easier, or (much more often) that some validation examples also appear in training. Check for leakage before you celebrate.\n\n## The thing that is not in the curve\n\nCatastrophic forgetting does not show up on either loss curve. Both are computed on data from your task distribution, and the capability you have destroyed is somewhere else entirely: the other language your product supports, the ability to say \"I do not know\", the refusal that used to work.\n\nThis is the argument for running the canary set at every checkpoint rather than at the end. It costs a few minutes per checkpoint, and it is the only instrument that points at the damage while you can still choose a different checkpoint. A run that is evaluated only at the end gives you one artefact and no alternatives.\n\n> Figure: Task metric and canary, across ten checkpoints. Line chart of score on a held-out set against checkpoint, saved every 200 steps: Task metric (1, 0.61) (2, 0.74) (3, 0.82) (4, 0.86) (5, 0.88) (6, 0.89) (7, 0.9) (8, 0.9) (9, 0.91) (10, 0.91); Canary, general capability (1, 0.84) (2, 0.84) (3, 0.83) (4, 0.83) (5, 0.82) (6, 0.79) (7, 0.75) (8, 0.71) (9, 0.67) (10, 0.62). The checkpoint to ship is the last one before the canary moves: the fifth here, which is neither the last checkpoint nor the best task score. Neither loss curve would have shown you this."},"check":{"question":"A run shows training loss falling steadily, validation loss rising from step 600, and routing accuracy on a held-out set still climbing at step 1,000. What should the team do?","options":[{"text":"Stop at step 600, since rising validation loss is the definition of overfitting and later checkpoints cannot be trusted.","correct":false,"tempting":"Applies the textbook rule mechanically; validation loss rises partly because the model has become more confident, which is not itself harm."},{"text":"Lower the learning rate so validation loss resumes falling, then continue to step 1,000.","correct":false,"tempting":"Treats the loss number as the objective and tunes to make it look right, rather than asking what the model is doing."},{"text":"Continue, but check a canary set at each checkpoint, since rising confidence raises loss while the metric improves.","correct":true},{"text":"Add more validation data, since a rising validation loss with an improving metric indicates the validation set is too small to be reliable.","correct":false,"tempting":"Attributes a systematic divergence between proxy and metric to sampling noise, which a larger validation set would not remove."}]},"prev":{"slug":"where-the-memory-goes","title":"Where the memory actually goes","url":"https://addaly.com/learn/fine-tuning/where-the-memory-goes"},"next":{"slug":"regularisation-that-helps","title":"Regularisation that helps, and the kind that does not","url":"https://addaly.com/learn/fine-tuning/regularisation-that-helps"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Epochs, overfitting and checkpoint selection', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/epochs-and-the-loss-curve (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Epochs, overfitting and checkpoint selection","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"The step down in training loss at each epoch boundary is memorisation, not learning; select checkpoints on a task metric with a canary set beside it rather than on validation loss, because the best checkpoint is often the last one before general capability starts to slip.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Task metric and canary, across ten checkpoints","series":[{"label":"Task metric","points":[[1,0.61],[2,0.74],[3,0.82],[4,0.86],[5,0.88],[6,0.89],[7,0.9],[8,0.9],[9,0.91],[10,0.91]]},{"label":"Canary, general capability","points":[[1,0.84],[2,0.84],[3,0.83],[4,0.83],[5,0.82],[6,0.79],[7,0.75],[8,0.71],[9,0.67],[10,0.62]]}],"xLabel":"checkpoint, saved every 200 steps","yLabel":"score on a held-out set","caption":"The checkpoint to ship is the last one before the canary moves: the fifth here, which is neither the last checkpoint nor the best task score. Neither loss curve would have shown you this."},"alt":"Figure: Task metric and canary, across ten checkpoints. Line chart of score on a held-out set against checkpoint, saved every 200 steps: Task metric (1, 0.61) (2, 0.74) (3, 0.82) (4, 0.86) (5, 0.88) (6, 0.89) (7, 0.9) (8, 0.9) (9, 0.91) (10, 0.91); Canary, general capability (1, 0.84) (2, 0.84) (3, 0.83) (4, 0.83) (5, 0.82) (6, 0.79) (7, 0.75) (8, 0.71) (9, 0.67) (10, 0.62). The checkpoint to ship is the last one before the canary moves: the fifth here, which is neither the last checkpoint nor the best task score. Neither loss curve would have shown you this."}},"blocks":[{"t":"h2","id":"one-to-three-and-why","runs":[{"text":"One to three, and why"}]},{"t":"p","runs":[{"text":"The standard advice for supervised fine-tuning is one to three epochs, and unusually for standard advice it is well grounded."}]},{"t":"p","runs":[{"text":"An epoch is one pass over your dataset. On the second pass the model sees examples it has seen before, and a model with billions of parameters and a few thousand examples is entirely capable of memorising them. What it memorises is not the skill you wanted: it is the specific answers, including their specific wording and the specific mistakes in your annotation."}]},{"t":"p","runs":[{"text":"The signature is unmistakable once you have seen it: at each epoch boundary the training loss takes a "},{"text":"step down","b":true},{"text":", a sharp discontinuity rather than a smooth continuation. That step is the model recognising the data. It is not learning anything at that moment; it is recalling."}]},{"t":"h2","id":"but-small-datasets-need-steps","runs":[{"text":"But small datasets need steps"}]},{"t":"p","runs":[{"text":"There is a real tension here. With 800 examples at an effective batch of 16, one epoch is 50 optimiser steps. Fifty steps will not move a freshly-initialised adapter anywhere useful."}]},{"t":"p","runs":[{"text":"So the honest framing is not “how many epochs” but “how many optimiser steps do I need, and can I get them without repeating data too often”. Roughly:"}]},{"t":"ul","items":[[{"text":"Under 500 steps total, an adapter is usually undertrained."}],[{"text":"500 to 2,000 steps is the normal range for a task LoRA."}],[{"text":"If reaching 500 steps requires six epochs over 600 examples, the problem is that you have 600 examples. Either collect more, or accept a smaller effective batch to get more steps from the same data: the latter is a reasonable trade at this scale."}]]},{"t":"p","runs":[{"text":"LoRA tolerates repetition better than full fine-tuning does, because the low-rank constraint limits how precisely it can memorise. That is a real advantage of the method and not usually mentioned as one."}]},{"t":"h2","id":"checkpoint-selection-done-properly","runs":[{"text":"Checkpoint selection, done properly"}]},{"t":"p","runs":[{"text":"Save a checkpoint every few hundred steps. At the end, you have five or ten candidates, and the question is which to ship."}]},{"t":"p","runs":[{"text":"Do "},{"text":"not","b":true},{"text":" pick on validation loss. As the previous lesson noted, validation loss can rise while your task metric improves, because a more confident model is penalised harder on the items it still gets wrong. If you early-stop on loss you will systematically ship earlier, blander checkpoints."}]},{"t":"p","runs":[{"text":"Pick on your task metric, evaluated on a held-out set, with the canary set alongside it:"}]},{"t":"code","lang":"python","label":"Python","text":"for ckpt in checkpoints:\n    task   = evaluate(ckpt, held_out)       # what you are trying to improve\n    canary = evaluate(ckpt, general_set)    # what must not break\n    print(ckpt, round(task, 3), round(canary, 3))"},{"t":"p","runs":[{"text":"The shape you usually see: task climbs quickly then plateaus, canary is flat then starts falling. The checkpoint to ship is the last one before the canary moves, which is frequently "},{"text":"not","i":true},{"text":" the last checkpoint and frequently not the best task score either."}]},{"t":"h2","id":"reading-the-curve-honestly","runs":[{"text":"Reading the curve honestly"}]},{"t":"p","runs":[{"text":"A few readings that are actually diagnostic, beyond the epoch step:"}]},{"t":"ul","items":[[{"text":"Train and validation loss both falling, gap steady.","b":true},{"text":" Healthy. Keep going."}],[{"text":"Train falling, validation flat.","b":true},{"text":" Learning your data’s idiosyncrasies. Not yet harmful, but the marginal epoch is buying little."}],[{"text":"Train falling, validation rising, task metric rising.","b":true},{"text":" Not necessarily overfitting in the harmful sense. Check the canary before you panic."}],[{"text":"Train falling, validation rising, task metric flat or falling.","b":true},{"text":" Overfitting, straightforwardly. Stop, and go back a checkpoint or two."}],[{"text":"Loss lower on validation than on training.","b":true},{"text":" Usually means your validation set is easier, or (much more often) that some validation examples also appear in training. Check for leakage before you celebrate."}]]},{"t":"h2","id":"the-thing-that-is-not-in-the-curve","runs":[{"text":"The thing that is not in the curve"}]},{"t":"p","runs":[{"text":"Catastrophic forgetting does not show up on either loss curve. Both are computed on data from your task distribution, and the capability you have destroyed is somewhere else entirely: the other language your product supports, the ability to say “I do not know”, the refusal that used to work."}]},{"t":"p","runs":[{"text":"This is the argument for running the canary set at every checkpoint rather than at the end. It costs a few minutes per checkpoint, and it is the only instrument that points at the damage while you can still choose a different checkpoint. A run that is evaluated only at the end gives you one artefact and no alternatives."}]}],"check":{"question":[{"text":"A run shows training loss falling steadily, validation loss rising from step 600, and routing accuracy on a held-out set still climbing at step 1,000. What should the team do?"}],"options":[{"text":[{"text":"Stop at step 600, since rising validation loss is the definition of overfitting and later checkpoints cannot be trusted."}],"correct":false,"tempting":[{"text":"Applies the textbook rule mechanically; validation loss rises partly because the model has become more confident, which is not itself harm."}]},{"text":[{"text":"Lower the learning rate so validation loss resumes falling, then continue to step 1,000."}],"correct":false,"tempting":[{"text":"Treats the loss number as the objective and tunes to make it look right, rather than asking what the model is doing."}]},{"text":[{"text":"Continue, but check a canary set at each checkpoint, since rising confidence raises loss while the metric improves."}],"correct":true,"tempting":null},{"text":[{"text":"Add more validation data, since a rising validation loss with an improving metric indicates the validation set is too small to be reliable."}],"correct":false,"tempting":[{"text":"Attributes a systematic divergence between proxy and metric to sampling noise, which a larger validation set would not remove."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"where-the-memory-goes","title":"Where the memory actually goes","minutes":10},"next":{"slug":"regularisation-that-helps","title":"Regularisation that helps, and the kind that does not","minutes":8}}}