{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"reading-a-loss-curve","title":"Diagnosing a model from the shape of its loss curve","url":"https://addaly.com/learn/machine-learning-foundations/reading-a-loss-curve","position":14,"of":92,"minutes":8,"answer":"Plot training and validation loss on one axis every run: the gap diagnoses fit, the slope at the end diagnoses training length.","keyIdea":"Plot training and validation loss on one axis every run: the gap diagnoses fit, the slope at the end diagnoses training length, and NaN or a flat line diagnoses a bug rather than a modelling problem."},"body":{"format":"markdown","text":"## The plot you should make every time\n\nTwo lines, x-axis is epochs or steps, y-axis is loss. Training loss and validation loss on the same axes. This one plot answers more questions than any other artefact in machine learning, and it takes four lines of code.\n\n```python\nimport matplotlib.pyplot as plt\nplt.plot(history[\"train_loss\"], label=\"train\")\nplt.plot(history[\"val_loss\"], label=\"validation\")\nplt.legend(); plt.xlabel(\"epoch\"); plt.ylabel(\"loss\")\n```\n\nFor scikit-learn models that do not record a history, `HistGradientBoostingClassifier` exposes `validation_score_` per iteration, and `SGDClassifier` can be driven with `partial_fit` in a loop to build one manually. For anything in PyTorch you write it yourself, and you should.\n\n## Seven shapes, and what each one means\n\n**Both lines falling, still falling at the end.** Undertrained. Train longer. The single most common mistake in a first project is stopping here and concluding the model is weak.\n\n**Both fall, then flatten together with a small gap.** This is the healthy result. The remaining gap is normal; a gap of exactly zero is suspicious and usually means the model is too simple to have fitted anything specific.\n\n**Train falls, validation flattens and then rises.** Overfitting, in its textbook form. The point where validation turns is where you should have stopped, and `early_stopping` does exactly that automatically.\n\n> Figure: The textbook overfitting curve. Line chart of Loss against Epoch: Validation loss (0, 0.7) (2, 0.55) (4, 0.47) (6, 0.42) (8, 0.39) (10, 0.38) (12, 0.385) (14, 0.4) (16, 0.42) (18, 0.45) (20, 0.48); Training loss (0, 0.69) (2, 0.52) (4, 0.43) (6, 0.37) (8, 0.32) (10, 0.28) (12, 0.25) (14, 0.22) (16, 0.2) (18, 0.18) (20, 0.16). Training loss falls for as long as you let it. Validation loss bottoms out at epoch ten and rises from there, which is the model memorising this particular noise. The epoch where validation turns is where you should have stopped, and early_stopping does exactly that.\n\n**Both flat and high from step one.** Nothing is learning. Check that the learning rate is not absurdly small, that the features actually reach the model, and that the labels are not shuffled relative to the features. A model that predicts the base rate forever usually has a plumbing fault, not a modelling one.\n\n**Loss becomes NaN.** Learning rate far too high, or a division by zero somewhere in the features, or a log of zero in a custom loss. Drop the learning rate by a factor of ten first; it is the cause about eight times out of ten.\n\n**Validation loss below training loss.** Surprising, and it has two innocent causes and one guilty one. Innocent: regularisation such as dropout is active during training and off during evaluation, so training is measured on a handicapped model; and training loss is often averaged over the epoch while validation is measured at the end, so training looks worse by the amount the model improved during that epoch. Guilty: your validation set is easier than your training set, which happens when the split is not random.\n\n**A sudden cliff downward.** Usually a learning rate schedule stepping down, as described in the previous lesson. Sometimes an unfreezing event in a fine-tuning run. Rarely a real discovery.\n\n## The noise is information too\n\nA validation curve that jumps around wildly from epoch to epoch is telling you the validation set is small. The measurement noise on a set of 200 rows is large; with 5,000 rows the same model gives a smooth line. If your curve looks like static, do not tune against individual points: you will be chasing noise, and you will pick the epoch that got lucky.\n\nThis matters because early stopping picks the minimum of a noisy curve, which is biased low. That is fine for choosing when to stop and misleading if you then report that minimum as your score. Report the test set, once, afterwards.\n\n## Learning curve is a different plot\n\nWorth separating now because the names get used loosely. The plot above has *epochs* on the x-axis. A **learning curve** has *training set size* on the x-axis: fit on 10% of the data, then 20%, and so on, plotting train and validation score at each size.\n\nThat second plot answers a different and very expensive question: *would more data help?* If the validation curve is still climbing at 100% of your data, collecting more will help. If it flattened at 40%, more of the same data will not, and you need better features or a different model. We build it properly in module 4; the distinction is worth having now so the two are never confused.\n\n## Save the history\n\nA small habit with a large payoff: store the loss history alongside the model, with the hyperparameters, the seed, and the data version. Three weeks later someone asks why the model changed, and the answer is almost always visible in the two curves side by side. Reconstructing it from memory is not possible, and re-running it is often not either, because the data has moved on."},"check":{"question":"A neural network's validation loss sits consistently *below* its training loss for every epoch. The split was random and stratified. What is the most likely explanation?","options":[{"text":"The validation set is contaminated with training rows, making it artificially easy.","correct":false,"tempting":"Leakage would inflate validation performance, but it usually shows as validation tracking training almost exactly rather than sitting systematically below it."},{"text":"Dropout is active during training and disabled at evaluation, so the training loss is measured on a deliberately handicapped model.","correct":true},{"text":"The model is underfitting, and underfitted models always show lower validation loss.","correct":false,"tempting":"Underfitting produces both losses high and close together, with no reason for validation to be the lower of the two."},{"text":"The validation set is smaller, so its loss is averaged over fewer rows and is therefore lower.","correct":false,"tempting":"Averaging over fewer rows increases variance but does not bias the mean downward, so it cannot produce a consistent gap in one direction."}]},"prev":{"slug":"batches-and-the-learning-rate","title":"Batches, epochs, and the two knobs that decide everything","url":"https://addaly.com/learn/machine-learning-foundations/batches-and-the-learning-rate"},"next":{"slug":"logistic-regression-from-odds","title":"Logistic regression, built from odds rather than assumed","url":"https://addaly.com/learn/machine-learning-foundations/logistic-regression-from-odds"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Diagnosing a model from the shape of its loss curve', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/reading-a-loss-curve (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Diagnosing a model from the shape of its loss curve","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"Plot training and validation loss on one axis every run: the gap diagnoses fit, the slope at the end diagnoses training length, and NaN or a flat line diagnoses a bug rather than a modelling problem.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The textbook overfitting curve","series":[{"label":"Validation loss","points":[[0,0.7],[2,0.55],[4,0.47],[6,0.42],[8,0.39],[10,0.38],[12,0.385],[14,0.4],[16,0.42],[18,0.45],[20,0.48]]},{"label":"Training loss","points":[[0,0.69],[2,0.52],[4,0.43],[6,0.37],[8,0.32],[10,0.28],[12,0.25],[14,0.22],[16,0.2],[18,0.18],[20,0.16]]}],"xLabel":"Epoch","yLabel":"Loss","caption":"Training loss falls for as long as you let it. Validation loss bottoms out at epoch ten and rises from there, which is the model memorising this particular noise. The epoch where validation turns is where you should have stopped, and early_stopping does exactly that."},"alt":"Figure: The textbook overfitting curve. Line chart of Loss against Epoch: Validation loss (0, 0.7) (2, 0.55) (4, 0.47) (6, 0.42) (8, 0.39) (10, 0.38) (12, 0.385) (14, 0.4) (16, 0.42) (18, 0.45) (20, 0.48); Training loss (0, 0.69) (2, 0.52) (4, 0.43) (6, 0.37) (8, 0.32) (10, 0.28) (12, 0.25) (14, 0.22) (16, 0.2) (18, 0.18) (20, 0.16). Training loss falls for as long as you let it. Validation loss bottoms out at epoch ten and rises from there, which is the model memorising this particular noise. The epoch where validation turns is where you should have stopped, and early_stopping does exactly that."}},"blocks":[{"t":"h2","id":"the-plot-you-should-make-every-time","runs":[{"text":"The plot you should make every time"}]},{"t":"p","runs":[{"text":"Two lines, x-axis is epochs or steps, y-axis is loss. Training loss and validation loss on the same axes. This one plot answers more questions than any other artefact in machine learning, and it takes four lines of code."}]},{"t":"code","lang":"python","label":"Python","text":"import matplotlib.pyplot as plt\nplt.plot(history[\"train_loss\"], label=\"train\")\nplt.plot(history[\"val_loss\"], label=\"validation\")\nplt.legend(); plt.xlabel(\"epoch\"); plt.ylabel(\"loss\")"},{"t":"p","runs":[{"text":"For scikit-learn models that do not record a history, "},{"text":"HistGradientBoostingClassifier","code":true},{"text":" exposes "},{"text":"validation_score_","code":true},{"text":" per iteration, and "},{"text":"SGDClassifier","code":true},{"text":" can be driven with "},{"text":"partial_fit","code":true},{"text":" in a loop to build one manually. For anything in PyTorch you write it yourself, and you should."}]},{"t":"h2","id":"seven-shapes-and-what-each-one-means","runs":[{"text":"Seven shapes, and what each one means"}]},{"t":"p","runs":[{"text":"Both lines falling, still falling at the end.","b":true},{"text":" Undertrained. Train longer. The single most common mistake in a first project is stopping here and concluding the model is weak."}]},{"t":"p","runs":[{"text":"Both fall, then flatten together with a small gap.","b":true},{"text":" This is the healthy result. The remaining gap is normal; a gap of exactly zero is suspicious and usually means the model is too simple to have fitted anything specific."}]},{"t":"p","runs":[{"text":"Train falls, validation flattens and then rises.","b":true},{"text":" Overfitting, in its textbook form. The point where validation turns is where you should have stopped, and "},{"text":"early_stopping","code":true},{"text":" does exactly that automatically."}]},{"t":"p","runs":[{"text":"Both flat and high from step one.","b":true},{"text":" Nothing is learning. Check that the learning rate is not absurdly small, that the features actually reach the model, and that the labels are not shuffled relative to the features. A model that predicts the base rate forever usually has a plumbing fault, not a modelling one."}]},{"t":"p","runs":[{"text":"Loss becomes NaN.","b":true},{"text":" Learning rate far too high, or a division by zero somewhere in the features, or a log of zero in a custom loss. Drop the learning rate by a factor of ten first; it is the cause about eight times out of ten."}]},{"t":"p","runs":[{"text":"Validation loss below training loss.","b":true},{"text":" Surprising, and it has two innocent causes and one guilty one. Innocent: regularisation such as dropout is active during training and off during evaluation, so training is measured on a handicapped model; and training loss is often averaged over the epoch while validation is measured at the end, so training looks worse by the amount the model improved during that epoch. Guilty: your validation set is easier than your training set, which happens when the split is not random."}]},{"t":"p","runs":[{"text":"A sudden cliff downward.","b":true},{"text":" Usually a learning rate schedule stepping down, as described in the previous lesson. Sometimes an unfreezing event in a fine-tuning run. Rarely a real discovery."}]},{"t":"h2","id":"the-noise-is-information-too","runs":[{"text":"The noise is information too"}]},{"t":"p","runs":[{"text":"A validation curve that jumps around wildly from epoch to epoch is telling you the validation set is small. The measurement noise on a set of 200 rows is large; with 5,000 rows the same model gives a smooth line. If your curve looks like static, do not tune against individual points: you will be chasing noise, and you will pick the epoch that got lucky."}]},{"t":"p","runs":[{"text":"This matters because early stopping picks the minimum of a noisy curve, which is biased low. That is fine for choosing when to stop and misleading if you then report that minimum as your score. Report the test set, once, afterwards."}]},{"t":"h2","id":"learning-curve-is-a-different-plot","runs":[{"text":"Learning curve is a different plot"}]},{"t":"p","runs":[{"text":"Worth separating now because the names get used loosely. The plot above has "},{"text":"epochs","i":true},{"text":" on the x-axis. A "},{"text":"learning curve","b":true},{"text":" has "},{"text":"training set size","i":true},{"text":" on the x-axis: fit on 10% of the data, then 20%, and so on, plotting train and validation score at each size."}]},{"t":"p","runs":[{"text":"That second plot answers a different and very expensive question: "},{"text":"would more data help?","i":true},{"text":" If the validation curve is still climbing at 100% of your data, collecting more will help. If it flattened at 40%, more of the same data will not, and you need better features or a different model. We build it properly in module 4; the distinction is worth having now so the two are never confused."}]},{"t":"h2","id":"save-the-history","runs":[{"text":"Save the history"}]},{"t":"p","runs":[{"text":"A small habit with a large payoff: store the loss history alongside the model, with the hyperparameters, the seed, and the data version. Three weeks later someone asks why the model changed, and the answer is almost always visible in the two curves side by side. Reconstructing it from memory is not possible, and re-running it is often not either, because the data has moved on."}]}],"check":{"question":[{"text":"A neural network’s validation loss sits consistently "},{"text":"below","i":true},{"text":" its training loss for every epoch. The split was random and stratified. What is the most likely explanation?"}],"options":[{"text":[{"text":"The validation set is contaminated with training rows, making it artificially easy."}],"correct":false,"tempting":[{"text":"Leakage would inflate validation performance, but it usually shows as validation tracking training almost exactly rather than sitting systematically below it."}]},{"text":[{"text":"Dropout is active during training and disabled at evaluation, so the training loss is measured on a deliberately handicapped model."}],"correct":true,"tempting":null},{"text":[{"text":"The model is underfitting, and underfitted models always show lower validation loss."}],"correct":false,"tempting":[{"text":"Underfitting produces both losses high and close together, with no reason for validation to be the lower of the two."}]},{"text":[{"text":"The validation set is smaller, so its loss is averaged over fewer rows and is therefore lower."}],"correct":false,"tempting":[{"text":"Averaging over fewer rows increases variance but does not bias the mean downward, so it cannot produce a consistent gap in one direction."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"batches-and-the-learning-rate","title":"Batches, epochs, and the two knobs that decide everything","minutes":9},"next":{"slug":"logistic-regression-from-odds","title":"Logistic regression, built from odds rather than assumed","minutes":10}}}