PCA: fewer columns, and what you gave up

The idea

Find the direction in which the data varies most. Call it the first principal component. Then find the direction of greatest remaining variation that is perpendicular to the first. Repeat.

The result is a new set of axes, ordered by how much of the data’s variance each one carries. Keep the first few and you have compressed many columns into a few while losing as little variance as possible.

from sklearn.decomposition import PCA
p = PCA(n_components=0.95).fit(X_scaled)     # keep 95% of variance
p.n_components_                               # how many that took
p.explained_variance_ratio_[:5]

Passing a float between 0 and 1 lets PCA choose the number of components for you, which is usually what you want. “38 components retain 95% of the variance” is a more useful statement than “we kept 10”.

Two prerequisites

Standardise first. PCA maximises variance, and a feature measured in rupees has more variance than one in years for reasons that have nothing to do with importance. Without scaling, the first component is whatever feature has the largest units. This is the most common PCA mistake.

PCA finds linear structure only. It fits a rotated set of straight axes. Data on a curved surface (a spiral, a circle) is not compressed well by any linear projection, and PCA will report that it needs many components for something a human can see is one-dimensional. Kernel PCA and the neighbour-embedding methods in the next lesson exist for that case.

Reading the output

Three things are worth looking at every time.

The cumulative explained variance curve. Plot the cumulative sum of explained_variance_ratio_. A sharp rise that flattens early means your features are highly redundant. A near-straight diagonal means they are largely independent and PCA will not help you much.

The loadings. p.components_[0] gives the weight of each original feature in the first component. This is how you interpret what a component means. In survey data the first component is often “overall agreement”, because someone who answers positively to one item tends to answer positively to all. In financial data it is often “size”.

The reconstruction error. p.inverse_transform(p.transform(X)) maps back to the original space. Comparing that with X shows exactly what was lost, per row. Rows with high reconstruction error are rows the components fail to describe, which is a serviceable anomaly detector in itself.

What PCA is good for

• Speeding things up. 500 correlated features to 40 components makes downstream training an order of magnitude faster with little accuracy loss.
• Removing collinearity. Components are orthogonal by construction, which fixes the coefficient instability from module 2 completely.
• Denoising. Small-variance components often capture measurement noise; dropping them can improve a downstream model.
• Visualisation. Two components, one scatter plot, and you can see whether the classes separate at all before building anything.
• Preprocessing for distance methods. Reducing to 30 dimensions before k-NN or clustering makes distances meaningful again.

What it costs you

Interpretability. Component 1 is 0.31 * income + 0.28 * tenure - 0.19 * complaints + ... across every feature. You can no longer say “income matters most”, and for a model that must be explained to a regulator this is often disqualifying.

Variance is not the same as usefulness. PCA has never seen your target. It is entirely possible for the signal you need to live in the 12th component, carrying 0.4% of the variance, and for you to have discarded it. This happens, and it is the main reason PCA is not a routine preprocessing step for supervised learning.

If you want a projection that knows about the target, linear discriminant analysis finds directions that separate the classes rather than directions of greatest variance. It is the supervised counterpart and is limited to n_classes - 1 components.

Two practical variants

TruncatedSVD works on sparse matrices without centring them, which PCA cannot do. On TF-IDF matrices this is called latent semantic analysis, and it was the standard way to get dense document vectors for two decades before neural embeddings.

IncrementalPCA processes the data in batches, so you can fit it on more data than fits in memory.

And put it in the pipeline

PCA learns the components from data, so it must learn them from training rows only. Fitting PCA on the full dataset before splitting is on the leakage list in module 3 for exactly this reason, and it is one of the more commonly missed entries on that list.