{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"learning-curves-and-more-data","title":"Would more data help? The curve that answers it","url":"https://addaly.com/learn/machine-learning-foundations/learning-curves-and-more-data","position":37,"of":92,"minutes":8,"answer":"Plot validation score against training-set size: a curve still climbing means collect more data, a converged pair at a poor score means the features or the model family are the limit.","keyIdea":"Plot validation score against training-set size: a curve still climbing means collect more data, a converged pair at a poor score means the features or the model family are the limit, and no learning curve can anticipate a feature you have not built."},"body":{"format":"markdown","text":"## An expensive question with a cheap answer\n\n\"Should we spend three months collecting more data?\" is one of the most consequential questions a team asks, and it is usually answered by intuition. It can be answered by a plot that takes twenty minutes.\n\nFit the model on 10% of your training data, then 20%, and so on, recording both training and validation score at each size.\n\n```python\nfrom sklearn.model_selection import learning_curve\nimport numpy as np\nsizes, train_s, val_s = learning_curve(\n    pipe, X, y, cv=5,\n    train_sizes=np.linspace(0.1, 1.0, 8), scoring=\"roc_auc\")\n```\n\nPlot the means of `train_s` and `val_s` against `sizes`.\n\n## Three shapes, three decisions\n\n**The validation curve is still climbing at 100%.** More data will help. The size of the remaining slope tells you roughly how much: if the last doubling bought two points, the next doubling will buy less than two, and you can extrapolate a rough answer to \"is 50,000 more rows worth it\".\n\n**The validation curve flattened at 40% and the training curve is far above it.** More of the same data will not help much. You have a variance problem that additional rows are no longer reducing, and the remedy is regularisation, fewer features, or a simpler model.\n\n**Both curves flattened and converged at a poor score.** Neither more data nor more regularisation will help. The model family cannot express the pattern, or the features do not contain it. Better features, or a different model family. This is the most common shape in a stalled project and the one people most often misdiagnose as needing more data.\n\n> Figure: Error against training-set size, still climbing. Line chart of Error, per cent against Training rows used, per cent of the data: Validation error (10, 30) (20, 24) (35, 21) (50, 19) (65, 18) (80, 17.3) (90, 16.8) (100, 16.4); Training error (10, 5) (20, 9) (35, 11) (50, 12.5) (65, 13.5) (80, 14) (90, 14.5) (100, 15). Validation error is still falling at 100 per cent of the data: the last doubling took nearly three points off, so more rows will help, and the next doubling will buy less than three. Training error rising is the model meeting reality, not the model getting worse.\n\n## Why the training curve falls\n\nA detail worth understanding rather than memorising. The training score usually *decreases* as the training set grows, and beginners often read this as the model getting worse.\n\nIt is not. With 50 rows a flexible model can fit them nearly perfectly, so training error is near zero. With 50,000 rows it cannot fit them all, so training error rises to something realistic. The training curve falling and the validation curve rising, converging towards each other, is exactly the picture of a model becoming better calibrated to reality.\n\n## What the curve cannot tell you\n\nThree limits, stated because this plot is often over-read.\n\n**It assumes more data of the same kind.** If your existing rows are biased (the selection problems from module 3) then more of the same will not fix a thing, and the curve will happily suggest that it might.\n\n**It cannot see a new feature.** A learning curve that has flattened says nothing about whether adding a `days_since_last_purchase` column would change everything. Feature improvements move the whole curve upward, and the plot has no way to anticipate that.\n\n**Extrapolating far beyond your current size is unreliable.** Doubling is a defensible extrapolation. Predicting what ten times the data buys is not, because the curve's shape is empirical and the tail is exactly the part you have not measured.\n\n## Scaling laws, and their limits\n\nFor large models on large corpora, the relationship between data, parameters, compute and loss has been studied enough to produce fitted power laws (the Kaplan and Chinchilla lines of work) that predict loss surprisingly well over several orders of magnitude, and that changed how large models are budgeted.\n\nTwo honest caveats before anyone applies that here. Those laws were fitted for a specific setting: next-token prediction with transformers on text at very large scale. They describe *loss*, which is not the same as the accuracy of a downstream task, and the mapping between the two is not a smooth one. And they say nothing about a gradient boosting model on 80,000 rows of tabular data, where the practical returns to more data flatten far sooner.\n\nFor the work in this course, the learning curve on your own data is a better instrument than any general law.\n\n## The cheaper version\n\nIf `learning_curve` is too slow on your dataset, do it manually at three sizes (25%, 50%, 100%) and look at the direction. Three points is enough to distinguish \"still climbing\" from \"flat\", which is the decision you actually need to make. The eight-point version is nicer to look at and rarely changes the answer."},"check":{"question":"A learning curve shows training and validation scores converging at 0.71 by 40% of the data and staying flat to 100%. What should the team do next?","options":[{"text":"Collect more data, since the flat region indicates the model has not yet seen enough variety.","correct":false,"tempting":"Reads a flat curve as unfinished; convergence at a stable level is precisely the signal that additional rows of the same kind will not move the score."},{"text":"Add regularisation, since the two curves meeting indicates the model is overfitting.","correct":false,"tempting":"Inverts the diagnosis: overfitting shows as a persistent gap between the curves, and the curves here have converged."},{"text":"Build better features or move to a more expressive model family, since both curves have converged at a low score, indicating the current setup cannot express the pattern.","correct":true},{"text":"Increase the number of cross-validation folds to get a more reliable estimate before deciding.","correct":false,"tempting":"More folds tighten the estimate slightly but would not change a clearly converged pattern, and the decision does not hinge on that precision."}]},"prev":{"slug":"bias-and-variance","title":"Bias and variance, with numbers attached","url":"https://addaly.com/learn/machine-learning-foundations/bias-and-variance"},"next":{"slug":"hyperparameter-search","title":"Searching hyperparameters without fooling yourself","url":"https://addaly.com/learn/machine-learning-foundations/hyperparameter-search"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Would more data help? The curve that answers it', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/learning-curves-and-more-data (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Would more data help? The curve that answers it","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"Plot validation score against training-set size: a curve still climbing means collect more data, a converged pair at a poor score means the features or the model family are the limit, and no learning curve can anticipate a feature you have not built.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Error against training-set size, still climbing","series":[{"label":"Validation error","points":[[10,30],[20,24],[35,21],[50,19],[65,18],[80,17.3],[90,16.8],[100,16.4]]},{"label":"Training error","points":[[10,5],[20,9],[35,11],[50,12.5],[65,13.5],[80,14],[90,14.5],[100,15]]}],"xLabel":"Training rows used, per cent of the data","yLabel":"Error, per cent","caption":"Validation error is still falling at 100 per cent of the data: the last doubling took nearly three points off, so more rows will help, and the next doubling will buy less than three. Training error rising is the model meeting reality, not the model getting worse."},"alt":"Figure: Error against training-set size, still climbing. Line chart of Error, per cent against Training rows used, per cent of the data: Validation error (10, 30) (20, 24) (35, 21) (50, 19) (65, 18) (80, 17.3) (90, 16.8) (100, 16.4); Training error (10, 5) (20, 9) (35, 11) (50, 12.5) (65, 13.5) (80, 14) (90, 14.5) (100, 15). Validation error is still falling at 100 per cent of the data: the last doubling took nearly three points off, so more rows will help, and the next doubling will buy less than three. Training error rising is the model meeting reality, not the model getting worse."}},"blocks":[{"t":"h2","id":"an-expensive-question-with-a-cheap-answer","runs":[{"text":"An expensive question with a cheap answer"}]},{"t":"p","runs":[{"text":"“Should we spend three months collecting more data?” is one of the most consequential questions a team asks, and it is usually answered by intuition. It can be answered by a plot that takes twenty minutes."}]},{"t":"p","runs":[{"text":"Fit the model on 10% of your training data, then 20%, and so on, recording both training and validation score at each size."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import learning_curve\nimport numpy as np\nsizes, train_s, val_s = learning_curve(\n    pipe, X, y, cv=5,\n    train_sizes=np.linspace(0.1, 1.0, 8), scoring=\"roc_auc\")"},{"t":"p","runs":[{"text":"Plot the means of "},{"text":"train_s","code":true},{"text":" and "},{"text":"val_s","code":true},{"text":" against "},{"text":"sizes","code":true},{"text":"."}]},{"t":"h2","id":"three-shapes-three-decisions","runs":[{"text":"Three shapes, three decisions"}]},{"t":"p","runs":[{"text":"The validation curve is still climbing at 100%.","b":true},{"text":" More data will help. The size of the remaining slope tells you roughly how much: if the last doubling bought two points, the next doubling will buy less than two, and you can extrapolate a rough answer to “is 50,000 more rows worth it”."}]},{"t":"p","runs":[{"text":"The validation curve flattened at 40% and the training curve is far above it.","b":true},{"text":" More of the same data will not help much. You have a variance problem that additional rows are no longer reducing, and the remedy is regularisation, fewer features, or a simpler model."}]},{"t":"p","runs":[{"text":"Both curves flattened and converged at a poor score.","b":true},{"text":" Neither more data nor more regularisation will help. The model family cannot express the pattern, or the features do not contain it. Better features, or a different model family. This is the most common shape in a stalled project and the one people most often misdiagnose as needing more data."}]},{"t":"h2","id":"why-the-training-curve-falls","runs":[{"text":"Why the training curve falls"}]},{"t":"p","runs":[{"text":"A detail worth understanding rather than memorising. The training score usually "},{"text":"decreases","i":true},{"text":" as the training set grows, and beginners often read this as the model getting worse."}]},{"t":"p","runs":[{"text":"It is not. With 50 rows a flexible model can fit them nearly perfectly, so training error is near zero. With 50,000 rows it cannot fit them all, so training error rises to something realistic. The training curve falling and the validation curve rising, converging towards each other, is exactly the picture of a model becoming better calibrated to reality."}]},{"t":"h2","id":"what-the-curve-cannot-tell-you","runs":[{"text":"What the curve cannot tell you"}]},{"t":"p","runs":[{"text":"Three limits, stated because this plot is often over-read."}]},{"t":"p","runs":[{"text":"It assumes more data of the same kind.","b":true},{"text":" If your existing rows are biased (the selection problems from module 3) then more of the same will not fix a thing, and the curve will happily suggest that it might."}]},{"t":"p","runs":[{"text":"It cannot see a new feature.","b":true},{"text":" A learning curve that has flattened says nothing about whether adding a "},{"text":"days_since_last_purchase","code":true},{"text":" column would change everything. Feature improvements move the whole curve upward, and the plot has no way to anticipate that."}]},{"t":"p","runs":[{"text":"Extrapolating far beyond your current size is unreliable.","b":true},{"text":" Doubling is a defensible extrapolation. Predicting what ten times the data buys is not, because the curve’s shape is empirical and the tail is exactly the part you have not measured."}]},{"t":"h2","id":"scaling-laws-and-their-limits","runs":[{"text":"Scaling laws, and their limits"}]},{"t":"p","runs":[{"text":"For large models on large corpora, the relationship between data, parameters, compute and loss has been studied enough to produce fitted power laws (the Kaplan and Chinchilla lines of work) that predict loss surprisingly well over several orders of magnitude, and that changed how large models are budgeted."}]},{"t":"p","runs":[{"text":"Two honest caveats before anyone applies that here. Those laws were fitted for a specific setting: next-token prediction with transformers on text at very large scale. They describe "},{"text":"loss","i":true},{"text":", which is not the same as the accuracy of a downstream task, and the mapping between the two is not a smooth one. And they say nothing about a gradient boosting model on 80,000 rows of tabular data, where the practical returns to more data flatten far sooner."}]},{"t":"p","runs":[{"text":"For the work in this course, the learning curve on your own data is a better instrument than any general law."}]},{"t":"h2","id":"the-cheaper-version","runs":[{"text":"The cheaper version"}]},{"t":"p","runs":[{"text":"If "},{"text":"learning_curve","code":true},{"text":" is too slow on your dataset, do it manually at three sizes (25%, 50%, 100%) and look at the direction. Three points is enough to distinguish “still climbing” from “flat”, which is the decision you actually need to make. The eight-point version is nicer to look at and rarely changes the answer."}]}],"check":{"question":[{"text":"A learning curve shows training and validation scores converging at 0.71 by 40% of the data and staying flat to 100%. What should the team do next?"}],"options":[{"text":[{"text":"Collect more data, since the flat region indicates the model has not yet seen enough variety."}],"correct":false,"tempting":[{"text":"Reads a flat curve as unfinished; convergence at a stable level is precisely the signal that additional rows of the same kind will not move the score."}]},{"text":[{"text":"Add regularisation, since the two curves meeting indicates the model is overfitting."}],"correct":false,"tempting":[{"text":"Inverts the diagnosis: overfitting shows as a persistent gap between the curves, and the curves here have converged."}]},{"text":[{"text":"Build better features or move to a more expressive model family, since both curves have converged at a low score, indicating the current setup cannot express the pattern."}],"correct":true,"tempting":null},{"text":[{"text":"Increase the number of cross-validation folds to get a more reliable estimate before deciding."}],"correct":false,"tempting":[{"text":"More folds tighten the estimate slightly but would not change a clearly converged pattern, and the decision does not hinge on that precision."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"bias-and-variance","title":"Bias and variance, with numbers attached","minutes":9},"next":{"slug":"hyperparameter-search","title":"Searching hyperparameters without fooling yourself","minutes":9}}}