{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"overfitting-and-underfitting","title":"Overfitting, underfitting, and how to tell","url":"https://addaly.com/learn/machine-learning-foundations/overfitting-and-underfitting","position":33,"of":92,"minutes":8,"answer":"Compare training error with validation error; the gap between them, not the score itself, names the problem.","keyIdea":"Compare training error with validation error; the gap between them, not the score itself, names the problem."},"body":{"format":"markdown","text":"## Two ways to fail\n\nA student who understands the subject does well on a new exam. A student who memorised last year's paper does well on last year's paper. A student who skimmed the syllabus does badly on both.\n\nModels fail in exactly these two directions.\n\n**Overfitting**: the model has captured accidents of the training data: this particular noise, these particular rows. Training error is low, new-data error is much higher.\n\n**Underfitting**: the model is not expressive enough, or the features do not carry the signal. Both errors are high.\n\n## Capacity\n\nThe dial between them is **capacity**: how many distinguishable functions the model family can express.\n\nFit polynomials to 20 noisy points.\n\n- Degree 1, a straight line. Training error 0.31, validation error 0.33. It cannot follow the shape.\n- Degree 3. Training error 0.09, validation 0.11. About right.\n- Degree 15. Training error 0.002: the curve threads every point exactly. Validation error 0.94, worse than the straight line, because between the points it swings wildly.\n\nTraining error falls monotonically with capacity. Validation error falls, bottoms out, and then rises. That U is the whole story, and every capacity knob you meet is a way of moving along it: tree depth, number of trees, number of layers, number of features, how long you train.\n\n> Figure: Polynomials fitted to 20 noisy points. Line chart of Error against Polynomial degree: Validation error (1, 0.33) (3, 0.11) (6, 0.14) (9, 0.26) (12, 0.52) (15, 0.94); Training error (1, 0.31) (3, 0.09) (6, 0.05) (9, 0.03) (12, 0.01) (15, 0.002). Training error falls without limit as the degree rises. Validation error bottoms out near degree three and climbs to 0.94 at degree fifteen (worse than the straight line) because between the points the curve swings wildly. Every capacity knob you meet moves you along this U.\n\n## Diagnosis takes two numbers\n\nYou cannot diagnose from one score. You need training error and validation error side by side.\n\n| Train | Validation | Diagnosis | What to do |\n|---|---|---|---|\n| High | High, similar | Underfitting | More capacity, better features, train longer |\n| Low | Much higher | Overfitting | More data, less capacity, regularization, fewer features |\n| Low | Low | Working | Stop touching it |\n| High | Lower than train | Something is wrong | Check the split, check for duplicates |\n\nThat last row is not a joke. Validation beating training usually means leakage, a broken split, or dropout being left on during evaluation.\n\nThe common mistake is reading row one as row two. \"Overfitting\" has drifted in casual use to mean \"the model is bad\", so a disappointing score attracts the word regardless of the gap. If training accuracy is 71% and validation is 70%, nothing is being memorised. The model is not seeing enough, and cutting capacity will make it worse.\n\n## The cures, in order of how well they work\n\n**More data.** The most reliable fix and usually the least available. Doubling the rows moves overfitting more than any hyperparameter will.\n\n**Less capacity.** Shallower trees, fewer components, a smaller network.\n\n**Regularization**: a penalty added to the loss that discourages complexity:\n\n- *L2 (ridge)*: add the sum of squared weights to the loss. Weights shrink toward zero and no single feature dominates.\n- *L1 (lasso)*: add the sum of absolute weights. Weights are driven exactly to zero, which selects features as a side effect.\n- *Early stopping*: watch validation error each epoch and stop when it turns upward. Free, and it works.\n- *Dropout*: during training, randomly zero out some fraction of a network's units so no unit can rely on any other.\n- *Augmentation*: manufacture more training data by transforming what you have: flipping, cropping, adding noise. For images this is the strongest tool on the list.\n\nAll of these have a knob, and every knob is chosen on validation. Not on test.\n\n## An honest complication\n\nThe U-curve is the classical picture and it holds for the tabular models most people build. Very large neural networks break it. Push capacity far past the point of fitting the training data perfectly and validation error sometimes falls *again*: the phenomenon is called double descent, and a network with more parameters than training examples can still generalise well.\n\nWhy that happens is still argued over. It does not mean overfitting was a myth; it means capacity alone is a cruder predictor of generalisation than the textbook curve suggests. For a gradient-boosted model on 50,000 rows, keep using the U. For a 70-billion-parameter network, the intuitions in this lesson stop being reliable."},"check":{"question":"A gradient-boosted model reaches 71% accuracy on training data and 70% on validation. A colleague says it is overfitting and wants to cut the tree depth from 6 to 3. What does the evidence show?","options":[{"text":"It is underfitting: both numbers are low and the gap is negligible, so it needs more capacity or better features.","correct":true},{"text":"It is overfitting, since 70% is a weak result and overfitting is what makes results weak.","correct":false,"tempting":"\"Overfitting\" is used colloquially as a synonym for \"bad model\", so any disappointing score pulls the word toward it regardless of what the two numbers say."},{"text":"It is fitted well; a one-point gap is what a healthy model looks like.","correct":false,"tempting":"The tiny gap genuinely is healthy, and it is easy to slide from \"not overfitting\" to \"nothing wrong\" without noticing that both numbers are low."},{"text":"Nothing can be concluded until they check the test set.","correct":false,"tempting":"Deferring to the test set sounds like the disciplined answer, but the test set answers a different question and spending it on a diagnosis destroys the one honest estimate they have."}]},"prev":{"slug":"not-looking","title":"Train, validation, test, and the discipline of not looking","url":"https://addaly.com/learn/machine-learning-foundations/not-looking"},"next":{"slug":"cross-validation","title":"Cross-validation: using every row twice, legally","url":"https://addaly.com/learn/machine-learning-foundations/cross-validation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Overfitting, underfitting, and how to tell', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/overfitting-and-underfitting (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Overfitting, underfitting, and how to tell","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"Compare training error with validation error; the gap between them, not the score itself, names the problem.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Polynomials fitted to 20 noisy points","series":[{"label":"Validation error","points":[[1,0.33],[3,0.11],[6,0.14],[9,0.26],[12,0.52],[15,0.94]]},{"label":"Training error","points":[[1,0.31],[3,0.09],[6,0.05],[9,0.03],[12,0.01],[15,0.002]]}],"xLabel":"Polynomial degree","yLabel":"Error","caption":"Training error falls without limit as the degree rises. Validation error bottoms out near degree three and climbs to 0.94 at degree fifteen (worse than the straight line) because between the points the curve swings wildly. Every capacity knob you meet moves you along this U."},"alt":"Figure: Polynomials fitted to 20 noisy points. Line chart of Error against Polynomial degree: Validation error (1, 0.33) (3, 0.11) (6, 0.14) (9, 0.26) (12, 0.52) (15, 0.94); Training error (1, 0.31) (3, 0.09) (6, 0.05) (9, 0.03) (12, 0.01) (15, 0.002). Training error falls without limit as the degree rises. Validation error bottoms out near degree three and climbs to 0.94 at degree fifteen (worse than the straight line) because between the points the curve swings wildly. Every capacity knob you meet moves you along this U."}},"blocks":[{"t":"h2","id":"two-ways-to-fail","runs":[{"text":"Two ways to fail"}]},{"t":"p","runs":[{"text":"A student who understands the subject does well on a new exam. A student who memorised last year’s paper does well on last year’s paper. A student who skimmed the syllabus does badly on both."}]},{"t":"p","runs":[{"text":"Models fail in exactly these two directions."}]},{"t":"p","runs":[{"text":"Overfitting","b":true},{"text":": the model has captured accidents of the training data: this particular noise, these particular rows. Training error is low, new-data error is much higher."}]},{"t":"p","runs":[{"text":"Underfitting","b":true},{"text":": the model is not expressive enough, or the features do not carry the signal. Both errors are high."}]},{"t":"h2","id":"capacity","runs":[{"text":"Capacity"}]},{"t":"p","runs":[{"text":"The dial between them is "},{"text":"capacity","b":true},{"text":": how many distinguishable functions the model family can express."}]},{"t":"p","runs":[{"text":"Fit polynomials to 20 noisy points."}]},{"t":"ul","items":[[{"text":"Degree 1, a straight line. Training error 0.31, validation error 0.33. It cannot follow the shape."}],[{"text":"Degree 3. Training error 0.09, validation 0.11. About right."}],[{"text":"Degree 15. Training error 0.002: the curve threads every point exactly. Validation error 0.94, worse than the straight line, because between the points it swings wildly."}]]},{"t":"p","runs":[{"text":"Training error falls monotonically with capacity. Validation error falls, bottoms out, and then rises. That U is the whole story, and every capacity knob you meet is a way of moving along it: tree depth, number of trees, number of layers, number of features, how long you train."}]},{"t":"h2","id":"diagnosis-takes-two-numbers","runs":[{"text":"Diagnosis takes two numbers"}]},{"t":"p","runs":[{"text":"You cannot diagnose from one score. You need training error and validation error side by side."}]},{"t":"table","head":[[{"text":"Train"}],[{"text":"Validation"}],[{"text":"Diagnosis"}],[{"text":"What to do"}]],"align":["left","left","left","left"],"rows":[[[{"text":"High"}],[{"text":"High, similar"}],[{"text":"Underfitting"}],[{"text":"More capacity, better features, train longer"}]],[[{"text":"Low"}],[{"text":"Much higher"}],[{"text":"Overfitting"}],[{"text":"More data, less capacity, regularization, fewer features"}]],[[{"text":"Low"}],[{"text":"Low"}],[{"text":"Working"}],[{"text":"Stop touching it"}]],[[{"text":"High"}],[{"text":"Lower than train"}],[{"text":"Something is wrong"}],[{"text":"Check the split, check for duplicates"}]]]},{"t":"p","runs":[{"text":"That last row is not a joke. Validation beating training usually means leakage, a broken split, or dropout being left on during evaluation."}]},{"t":"p","runs":[{"text":"The common mistake is reading row one as row two. “Overfitting” has drifted in casual use to mean “the model is bad”, so a disappointing score attracts the word regardless of the gap. If training accuracy is 71% and validation is 70%, nothing is being memorised. The model is not seeing enough, and cutting capacity will make it worse."}]},{"t":"h2","id":"the-cures-in-order-of-how-well-they-work","runs":[{"text":"The cures, in order of how well they work"}]},{"t":"p","runs":[{"text":"More data.","b":true},{"text":" The most reliable fix and usually the least available. Doubling the rows moves overfitting more than any hyperparameter will."}]},{"t":"p","runs":[{"text":"Less capacity.","b":true},{"text":" Shallower trees, fewer components, a smaller network."}]},{"t":"p","runs":[{"text":"Regularization","b":true},{"text":": a penalty added to the loss that discourages complexity:"}]},{"t":"ul","items":[[{"text":"L2 (ridge)","i":true},{"text":": add the sum of squared weights to the loss. Weights shrink toward zero and no single feature dominates."}],[{"text":"L1 (lasso)","i":true},{"text":": add the sum of absolute weights. Weights are driven exactly to zero, which selects features as a side effect."}],[{"text":"Early stopping","i":true},{"text":": watch validation error each epoch and stop when it turns upward. Free, and it works."}],[{"text":"Dropout","i":true},{"text":": during training, randomly zero out some fraction of a network’s units so no unit can rely on any other."}],[{"text":"Augmentation","i":true},{"text":": manufacture more training data by transforming what you have: flipping, cropping, adding noise. For images this is the strongest tool on the list."}]]},{"t":"p","runs":[{"text":"All of these have a knob, and every knob is chosen on validation. Not on test."}]},{"t":"h2","id":"an-honest-complication","runs":[{"text":"An honest complication"}]},{"t":"p","runs":[{"text":"The U-curve is the classical picture and it holds for the tabular models most people build. Very large neural networks break it. Push capacity far past the point of fitting the training data perfectly and validation error sometimes falls "},{"text":"again","i":true},{"text":": the phenomenon is called double descent, and a network with more parameters than training examples can still generalise well."}]},{"t":"p","runs":[{"text":"Why that happens is still argued over. It does not mean overfitting was a myth; it means capacity alone is a cruder predictor of generalisation than the textbook curve suggests. For a gradient-boosted model on 50,000 rows, keep using the U. For a 70-billion-parameter network, the intuitions in this lesson stop being reliable."}]}],"check":{"question":[{"text":"A gradient-boosted model reaches 71% accuracy on training data and 70% on validation. A colleague says it is overfitting and wants to cut the tree depth from 6 to 3. What does the evidence show?"}],"options":[{"text":[{"text":"It is underfitting: both numbers are low and the gap is negligible, so it needs more capacity or better features."}],"correct":true,"tempting":null},{"text":[{"text":"It is overfitting, since 70% is a weak result and overfitting is what makes results weak."}],"correct":false,"tempting":[{"text":"“Overfitting” is used colloquially as a synonym for “bad model”, so any disappointing score pulls the word toward it regardless of what the two numbers say."}]},{"text":[{"text":"It is fitted well; a one-point gap is what a healthy model looks like."}],"correct":false,"tempting":[{"text":"The tiny gap genuinely is healthy, and it is easy to slide from “not overfitting” to “nothing wrong” without noticing that both numbers are low."}]},{"text":[{"text":"Nothing can be concluded until they check the test set."}],"correct":false,"tempting":[{"text":"Deferring to the test set sounds like the disciplined answer, but the test set answers a different question and spending it on a diagnosis destroys the one honest estimate they have."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"not-looking","title":"Train, validation, test, and the discipline of not looking","minutes":9},"next":{"slug":"cross-validation","title":"Cross-validation: using every row twice, legally","minutes":9}}}