Would more data help? The curve that answers it

An expensive question with a cheap answer

“Should we spend three months collecting more data?” is one of the most consequential questions a team asks, and it is usually answered by intuition. It can be answered by a plot that takes twenty minutes.

Fit the model on 10% of your training data, then 20%, and so on, recording both training and validation score at each size.

from sklearn.model_selection import learning_curve
import numpy as np
sizes, train_s, val_s = learning_curve(
    pipe, X, y, cv=5,
    train_sizes=np.linspace(0.1, 1.0, 8), scoring="roc_auc")

Plot the means of train_s and val_s against sizes.

Three shapes, three decisions

The validation curve is still climbing at 100%. More data will help. The size of the remaining slope tells you roughly how much: if the last doubling bought two points, the next doubling will buy less than two, and you can extrapolate a rough answer to “is 50,000 more rows worth it”.

The validation curve flattened at 40% and the training curve is far above it. More of the same data will not help much. You have a variance problem that additional rows are no longer reducing, and the remedy is regularisation, fewer features, or a simpler model.

Both curves flattened and converged at a poor score. Neither more data nor more regularisation will help. The model family cannot express the pattern, or the features do not contain it. Better features, or a different model family. This is the most common shape in a stalled project and the one people most often misdiagnose as needing more data.

Why the training curve falls

A detail worth understanding rather than memorising. The training score usually decreases as the training set grows, and beginners often read this as the model getting worse.

It is not. With 50 rows a flexible model can fit them nearly perfectly, so training error is near zero. With 50,000 rows it cannot fit them all, so training error rises to something realistic. The training curve falling and the validation curve rising, converging towards each other, is exactly the picture of a model becoming better calibrated to reality.

What the curve cannot tell you

Three limits, stated because this plot is often over-read.

It assumes more data of the same kind. If your existing rows are biased (the selection problems from module 3) then more of the same will not fix a thing, and the curve will happily suggest that it might.

It cannot see a new feature. A learning curve that has flattened says nothing about whether adding a days_since_last_purchase column would change everything. Feature improvements move the whole curve upward, and the plot has no way to anticipate that.

Extrapolating far beyond your current size is unreliable. Doubling is a defensible extrapolation. Predicting what ten times the data buys is not, because the curve’s shape is empirical and the tail is exactly the part you have not measured.

Scaling laws, and their limits

For large models on large corpora, the relationship between data, parameters, compute and loss has been studied enough to produce fitted power laws (the Kaplan and Chinchilla lines of work) that predict loss surprisingly well over several orders of magnitude, and that changed how large models are budgeted.

Two honest caveats before anyone applies that here. Those laws were fitted for a specific setting: next-token prediction with transformers on text at very large scale. They describe loss, which is not the same as the accuracy of a downstream task, and the mapping between the two is not a smooth one. And they say nothing about a gradient boosting model on 80,000 rows of tabular data, where the practical returns to more data flatten far sooner.

For the work in this course, the learning curve on your own data is a better instrument than any general law.

The cheaper version

If learning_curve is too slow on your dataset, do it manually at three sizes (25%, 50%, 100%) and look at the direction. Three points is enough to distinguish “still climbing” from “flat”, which is the decision you actually need to make. The eight-point version is nicer to look at and rarely changes the answer.