Covariance, and the shape of a cloud of points

From one variable to two

Variance measures how much one quantity wobbles. Covariance measures whether two wobble together:

cov(X, Y) = average of (x - mean_x)(y - mean_y)

Positive when they tend to be above or below their means at the same time, negative when one is high while the other is low, near zero when there is no linear pattern.

Its units are the product of the two variables' units, which makes the raw number impossible to interpret — a covariance of 4,200 between height in centimetres and weight in kilograms tells you nothing on its own. Divide by both standard deviations and you get correlation, which is unitless and comparable. Covariance is the quantity the maths uses; correlation is the quantity people read.

The matrix

With d features you get a d × d matrix: variances down the diagonal, covariances off it. It is symmetric, since cov(X, Y) = cov(Y, X), and that symmetry is what makes everything downstream well behaved.

import numpy as np
X = np.random.randn(1000, 3)
X[:, 1] += 0.8 * X[:, 0]              # make features 0 and 1 dependent
C = np.cov(X, rowvar=False)
print(C.round(2))

The matrix is a complete description of the shape of the data cloud, provided the cloud is elliptical. It says how wide it is in each feature direction, and how tilted.

Eigenvectors of the covariance are the axes of the cloud

Here is where the linear algebra module pays off. Because the covariance matrix is symmetric, its eigenvectors are perpendicular and its eigenvalues are real and non-negative. Those eigenvectors are the principal axes of the data cloud, and each eigenvalue is the variance along its axis.

So a cloud of points shaped like a tilted cigar has one large eigenvalue, along the length of the cigar, and small ones across it. Sorting the eigenvalues and keeping the largest few is exactly principal component analysis, which gets a worked lesson later. The point here is that PCA is not a separate technique with its own theory. It is reading the eigenvectors of a covariance matrix.

The multivariate normal

The multivariate normal is the natural extension of the bell curve to several dimensions, and it is fully specified by two things: a mean vector, saying where the cloud sits, and a covariance matrix, saying its shape and orientation. Nothing else.

Three cases of the covariance matrix are worth being able to picture:

• Identity matrix. A perfectly round cloud, all directions equal, no correlation.
• Diagonal but unequal. An axis-aligned ellipse, stretched more in some feature directions than others. This is what "diagonal covariance" means when you see it as an assumption in a model, and it is a claim that the features are uncorrelated.
• Full matrix with off-diagonal terms. A tilted ellipse. The tilt is the correlation.

This is why a full covariance is expensive: it has d(d+1)/2 free parameters. For 768 dimensions that is 295,296 numbers to estimate, and estimating them well needs far more than 768 data points. That parameter count, not any deep principle, is why so many methods assume a diagonal covariance — it is the only version you can estimate from a realistic sample.

Whitening

If you multiply your data by the inverse square root of the covariance matrix, the result has an identity covariance: round, unit-scale, uncorrelated. This is called whitening, and it is worth knowing because it explains several things at once.

Standardising each column — subtract mean, divide by standard deviation — is the cheap diagonal version of whitening. It fixes the scale differences but leaves the tilt. Full whitening removes the tilt as well, and it is what makes the loss surface of a linear model well conditioned, so gradient descent stops zigzagging. The connection between the optimisation lesson and this one is direct: badly correlated features produce a badly conditioned problem, and whitening is the fix that removes the cause.

The catch is that inverting a covariance matrix estimated from too little data is exactly the ill-conditioned operation from the matrix module. Whitening a 768-dimensional dataset with 500 samples will amplify noise dramatically, because the smallest estimated eigenvalues are essentially random and you are dividing by their square roots.

Where you will actually meet it

• PCA and dimensionality reduction, as above.
• Mahalanobis distance, which measures distance in units of the data's own spread rather than raw units. It is the sensible distance to use for outlier detection when features have different scales and are correlated.
• Gaussian mixture models, which fit several multivariate normals and where the choice between full, diagonal and spherical covariance is the main capacity knob.
• Embedding analysis. The covariance of an embedding set tells you whether the model is using its dimensions evenly. A spectrum dominated by a few large eigenvalues means the embeddings occupy a narrow cone, which is the anisotropy mentioned in the module on vectors, and it is measured exactly this way.

The rule to keep

The covariance matrix is the shape of the cloud, its eigenvectors are the cloud's axes, and estimating a full one needs far more data than most people have. When in doubt, standardise, plot the first two principal components, and look.