Bias and variance, with numbers attached

The decomposition

Imagine fitting your model on many different samples of the same size from the same population, and asking about the expected squared error on one test point. It splits into three parts:

expected error = bias^2 + variance + irreducible noise

Bias is how far the average prediction across all those fits is from the truth. High bias means the model family cannot represent the pattern: a straight line trying to fit a curve. It is wrong the same way every time.

Variance is how much the prediction moves between fits. High variance means the model is sensitive to which rows it happened to see. It is wrong differently every time.

Irreducible noise is the part of the target no model can predict, because it is not determined by your features. This is the ceiling from module 1, and no method removes it.

Watching it happen

Fit polynomials of increasing degree to 20 noisy points from a curve, and refit each on 50 different samples of 20 points.

• Degree 1. All 50 lines are nearly identical (low variance) and all of them miss the curve in the same places. High bias.
• Degree 3. The 50 curves are close to the truth on average and reasonably close to each other. This is the sweet spot.
• Degree 15. Each of the 50 curves passes almost exactly through its own 20 points, and they are wildly different from each other. Average them and you get close to the truth (low bias) but any single one is far off. High variance.

The useful thing about this picture is that it names the fix. High bias: more capacity, better features, less regularisation. High variance: more data, more regularisation, less capacity, or averaging several models.

How to tell which you have

From the two numbers you already have.

• Training error high, validation error about the same and also high: bias. The model cannot even fit what it has seen.
• Training error low, validation error much higher: variance. It fitted what it saw and did not generalise.
• Both low, close together: you are done.
• Training error high, validation error lower: check your pipeline, this is usually a bug or an artefact of regularisation being active only during training.

One caution about “high” and “low”. They only mean anything relative to the irreducible noise. A training error of 12% on a task where two experts disagree 11% of the time is not high bias; it is a model at the ceiling. This is why estimating human agreement, from module 1, is worth the afternoon it costs.

Bagging and boosting, in one sentence each

The decomposition explains the two great ensemble families, which module 6 covers in full.

Bagging (random forests) fits many high-variance models on different bootstrap samples and averages them. Averaging reduces variance without changing bias, so it takes a family that is individually unstable and makes it stable. That is why the individual trees in a random forest are grown deep and unpruned: you want them high-variance, because averaging handles it.

Boosting fits many high-bias models in sequence, each correcting the previous one’s errors. It attacks bias, which is why the individual trees in a boosted model are shallow: depth 3 to 6, models that on their own are barely better than a coin.

Same decomposition, opposite ends of it.

Where the classical picture breaks

The textbook draws a U: error falls as complexity rises, reaches a minimum, then climbs as variance takes over. Choose the bottom of the U.

That picture is accurate for the models in this module and misleading for large neural networks, which routinely have far more parameters than training rows and should therefore be at the far-right catastrophic end of the U. They are not; they generalise well.

The empirical finding, named double descent, is that test error follows the U up to roughly the point where the model can exactly fit the training data, and then falls again as capacity increases further. This is not folklore; it has been demonstrated repeatedly since 2018 across model families, and it is not fully explained. The leading accounts involve implicit regularisation by the optimiser and the geometry of the solutions gradient descent happens to find, and they remain accounts rather than a settled theory.

What that means for you: the bias-variance frame is a good instrument for the models in this course and should not be treated as a law of nature. Anyone who tells you a model with more parameters than data must overfit is describing a picture that stopped being complete about eight years ago.