{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"principal-components","title":"PCA: fewer columns, and what you gave up","url":"https://addaly.com/learn/machine-learning-foundations/principal-components","position":66,"of":92,"minutes":9,"answer":"PCA rotates to axes ordered by variance and keeps the first few, which requires standardised features and finds only linear structure, and because it never sees the target.","keyIdea":"PCA rotates to axes ordered by variance and keeps the first few, which requires standardised features and finds only linear structure, and because it never sees the target, the variance it discards can be the signal you needed."},"body":{"format":"markdown","text":"## The idea\n\nFind the direction in which the data varies most. Call it the first principal component. Then find the direction of greatest remaining variation that is perpendicular to the first. Repeat.\n\nThe result is a new set of axes, ordered by how much of the data's variance each one carries. Keep the first few and you have compressed many columns into a few while losing as little variance as possible.\n\n```python\nfrom sklearn.decomposition import PCA\np = PCA(n_components=0.95).fit(X_scaled)     # keep 95% of variance\np.n_components_                               # how many that took\np.explained_variance_ratio_[:5]\n```\n\nPassing a float between 0 and 1 lets PCA choose the number of components for you, which is usually what you want. \"38 components retain 95% of the variance\" is a more useful statement than \"we kept 10\".\n\n## Two prerequisites\n\n**Standardise first.** PCA maximises variance, and a feature measured in rupees has more variance than one in years for reasons that have nothing to do with importance. Without scaling, the first component is whatever feature has the largest units. This is the most common PCA mistake.\n\n**PCA finds linear structure only.** It fits a rotated set of straight axes. Data on a curved surface (a spiral, a circle) is not compressed well by any linear projection, and PCA will report that it needs many components for something a human can see is one-dimensional. Kernel PCA and the neighbour-embedding methods in the next lesson exist for that case.\n\n## Reading the output\n\nThree things are worth looking at every time.\n\n**The cumulative explained variance curve.** Plot the cumulative sum of `explained_variance_ratio_`. A sharp rise that flattens early means your features are highly redundant. A near-straight diagonal means they are largely independent and PCA will not help you much.\n\n> Figure: Cumulative variance explained, two 100-column tables. Line chart of Cumulative variance explained against Components kept: Highly redundant features (1, 0.42) (2, 0.61) (5, 0.8) (10, 0.9) (20, 0.95) (40, 0.98) (60, 0.99) (100, 1); Largely independent features (1, 0.02) (10, 0.12) (20, 0.23) (40, 0.44) (60, 0.64) (80, 0.83) (100, 1). The redundant table reaches 95 per cent of its variance with 20 components; the independent one needs nearly all hundred, and PCA buys almost nothing. The axis is variance, not usefulness. The target was never consulted, and the signal you need can sit in a component this curve tells you to discard.\n\n**The loadings.** `p.components_[0]` gives the weight of each original feature in the first component. This is how you interpret what a component means. In survey data the first component is often \"overall agreement\", because someone who answers positively to one item tends to answer positively to all. In financial data it is often \"size\".\n\n**The reconstruction error.** `p.inverse_transform(p.transform(X))` maps back to the original space. Comparing that with `X` shows exactly what was lost, per row. Rows with high reconstruction error are rows the components fail to describe, which is a serviceable anomaly detector in itself.\n\n## What PCA is good for\n\n- **Speeding things up.** 500 correlated features to 40 components makes downstream training an order of magnitude faster with little accuracy loss.\n- **Removing collinearity.** Components are orthogonal by construction, which fixes the coefficient instability from module 2 completely.\n- **Denoising.** Small-variance components often capture measurement noise; dropping them can improve a downstream model.\n- **Visualisation.** Two components, one scatter plot, and you can see whether the classes separate at all before building anything.\n- **Preprocessing for distance methods.** Reducing to 30 dimensions before k-NN or clustering makes distances meaningful again.\n\n## What it costs you\n\n**Interpretability.** Component 1 is `0.31 * income + 0.28 * tenure - 0.19 * complaints + ...` across every feature. You can no longer say \"income matters most\", and for a model that must be explained to a regulator this is often disqualifying.\n\n**Variance is not the same as usefulness.** PCA has never seen your target. It is entirely possible for the signal you need to live in the 12th component, carrying 0.4% of the variance, and for you to have discarded it. This happens, and it is the main reason PCA is not a routine preprocessing step for supervised learning.\n\nIf you want a projection that knows about the target, **linear discriminant analysis** finds directions that separate the classes rather than directions of greatest variance. It is the supervised counterpart and is limited to `n_classes - 1` components.\n\n## Two practical variants\n\n**TruncatedSVD** works on sparse matrices without centring them, which PCA cannot do. On TF-IDF matrices this is called latent semantic analysis, and it was the standard way to get dense document vectors for two decades before neural embeddings.\n\n**IncrementalPCA** processes the data in batches, so you can fit it on more data than fits in memory.\n\n## And put it in the pipeline\n\nPCA learns the components from data, so it must learn them from training rows only. Fitting PCA on the full dataset before splitting is on the leakage list in module 3 for exactly this reason, and it is one of the more commonly missed entries on that list."},"check":{"question":"A team applies PCA to 200 features, keeps components covering 95% of variance, and finds their classifier's AUC drops from 0.86 to 0.79. What is the most likely explanation?","options":[{"text":"95% is too aggressive a threshold, and keeping 99% would always recover the lost performance.","correct":false,"tempting":"Raising the threshold retains more components and may help, but the mechanism (that variance and predictiveness are different things) persists at any threshold."},{"text":"PCA orders components by variance without reference to the target, so a discriminative direction carrying little variance can be among the 5% discarded.","correct":true},{"text":"PCA requires the target to be included in the matrix so the components align with the outcome.","correct":false,"tempting":"Including the target would leak it into the features; PCA is unsupervised by design."},{"text":"The components are orthogonal, which removes the feature interactions the classifier relied on.","correct":false,"tempting":"Orthogonality removes correlation between components, but a model can still form interactions among them, so this is not what caused the loss."}]},"prev":{"slug":"density-and-hierarchy","title":"DBSCAN and hierarchical clustering: when shape and structure matter","url":"https://addaly.com/learn/machine-learning-foundations/density-and-hierarchy"},"next":{"slug":"seeing-high-dimensional-data","title":"t-SNE and UMAP, and how they mislead","url":"https://addaly.com/learn/machine-learning-foundations/seeing-high-dimensional-data"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'PCA: fewer columns, and what you gave up', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/principal-components (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"PCA: fewer columns, and what you gave up","courseTitle":"Machine Learning, Foundations","module":{"index":7,"slug":"learning-without-labels","title":"Learning without labels, and learning a representation"},"answer":null,"keyIdea":"PCA rotates to axes ordered by variance and keeps the first few, which requires standardised features and finds only linear structure, and because it never sees the target, the variance it discards can be the signal you needed.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Cumulative variance explained, two 100-column tables","series":[{"label":"Highly redundant features","points":[[1,0.42],[2,0.61],[5,0.8],[10,0.9],[20,0.95],[40,0.98],[60,0.99],[100,1]]},{"label":"Largely independent features","points":[[1,0.02],[10,0.12],[20,0.23],[40,0.44],[60,0.64],[80,0.83],[100,1]]}],"xLabel":"Components kept","yLabel":"Cumulative variance explained","caption":"The redundant table reaches 95 per cent of its variance with 20 components; the independent one needs nearly all hundred, and PCA buys almost nothing. The axis is variance, not usefulness. The target was never consulted, and the signal you need can sit in a component this curve tells you to discard."},"alt":"Figure: Cumulative variance explained, two 100-column tables. Line chart of Cumulative variance explained against Components kept: Highly redundant features (1, 0.42) (2, 0.61) (5, 0.8) (10, 0.9) (20, 0.95) (40, 0.98) (60, 0.99) (100, 1); Largely independent features (1, 0.02) (10, 0.12) (20, 0.23) (40, 0.44) (60, 0.64) (80, 0.83) (100, 1). The redundant table reaches 95 per cent of its variance with 20 components; the independent one needs nearly all hundred, and PCA buys almost nothing. The axis is variance, not usefulness. The target was never consulted, and the signal you need can sit in a component this curve tells you to discard."}},"blocks":[{"t":"h2","id":"the-idea","runs":[{"text":"The idea"}]},{"t":"p","runs":[{"text":"Find the direction in which the data varies most. Call it the first principal component. Then find the direction of greatest remaining variation that is perpendicular to the first. Repeat."}]},{"t":"p","runs":[{"text":"The result is a new set of axes, ordered by how much of the data’s variance each one carries. Keep the first few and you have compressed many columns into a few while losing as little variance as possible."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.decomposition import PCA\np = PCA(n_components=0.95).fit(X_scaled)     # keep 95% of variance\np.n_components_                               # how many that took\np.explained_variance_ratio_[:5]"},{"t":"p","runs":[{"text":"Passing a float between 0 and 1 lets PCA choose the number of components for you, which is usually what you want. “38 components retain 95% of the variance” is a more useful statement than “we kept 10”."}]},{"t":"h2","id":"two-prerequisites","runs":[{"text":"Two prerequisites"}]},{"t":"p","runs":[{"text":"Standardise first.","b":true},{"text":" PCA maximises variance, and a feature measured in rupees has more variance than one in years for reasons that have nothing to do with importance. Without scaling, the first component is whatever feature has the largest units. This is the most common PCA mistake."}]},{"t":"p","runs":[{"text":"PCA finds linear structure only.","b":true},{"text":" It fits a rotated set of straight axes. Data on a curved surface (a spiral, a circle) is not compressed well by any linear projection, and PCA will report that it needs many components for something a human can see is one-dimensional. Kernel PCA and the neighbour-embedding methods in the next lesson exist for that case."}]},{"t":"h2","id":"reading-the-output","runs":[{"text":"Reading the output"}]},{"t":"p","runs":[{"text":"Three things are worth looking at every time."}]},{"t":"p","runs":[{"text":"The cumulative explained variance curve.","b":true},{"text":" Plot the cumulative sum of "},{"text":"explained_variance_ratio_","code":true},{"text":". A sharp rise that flattens early means your features are highly redundant. A near-straight diagonal means they are largely independent and PCA will not help you much."}]},{"t":"p","runs":[{"text":"The loadings.","b":true},{"text":" "},{"text":"p.components_[0]","code":true},{"text":" gives the weight of each original feature in the first component. This is how you interpret what a component means. In survey data the first component is often “overall agreement”, because someone who answers positively to one item tends to answer positively to all. In financial data it is often “size”."}]},{"t":"p","runs":[{"text":"The reconstruction error.","b":true},{"text":" "},{"text":"p.inverse_transform(p.transform(X))","code":true},{"text":" maps back to the original space. Comparing that with "},{"text":"X","code":true},{"text":" shows exactly what was lost, per row. Rows with high reconstruction error are rows the components fail to describe, which is a serviceable anomaly detector in itself."}]},{"t":"h2","id":"what-pca-is-good-for","runs":[{"text":"What PCA is good for"}]},{"t":"ul","items":[[{"text":"Speeding things up.","b":true},{"text":" 500 correlated features to 40 components makes downstream training an order of magnitude faster with little accuracy loss."}],[{"text":"Removing collinearity.","b":true},{"text":" Components are orthogonal by construction, which fixes the coefficient instability from module 2 completely."}],[{"text":"Denoising.","b":true},{"text":" Small-variance components often capture measurement noise; dropping them can improve a downstream model."}],[{"text":"Visualisation.","b":true},{"text":" Two components, one scatter plot, and you can see whether the classes separate at all before building anything."}],[{"text":"Preprocessing for distance methods.","b":true},{"text":" Reducing to 30 dimensions before k-NN or clustering makes distances meaningful again."}]]},{"t":"h2","id":"what-it-costs-you","runs":[{"text":"What it costs you"}]},{"t":"p","runs":[{"text":"Interpretability.","b":true},{"text":" Component 1 is "},{"text":"0.31 * income + 0.28 * tenure - 0.19 * complaints + ...","code":true},{"text":" across every feature. You can no longer say “income matters most”, and for a model that must be explained to a regulator this is often disqualifying."}]},{"t":"p","runs":[{"text":"Variance is not the same as usefulness.","b":true},{"text":" PCA has never seen your target. It is entirely possible for the signal you need to live in the 12th component, carrying 0.4% of the variance, and for you to have discarded it. This happens, and it is the main reason PCA is not a routine preprocessing step for supervised learning."}]},{"t":"p","runs":[{"text":"If you want a projection that knows about the target, "},{"text":"linear discriminant analysis","b":true},{"text":" finds directions that separate the classes rather than directions of greatest variance. It is the supervised counterpart and is limited to "},{"text":"n_classes - 1","code":true},{"text":" components."}]},{"t":"h2","id":"two-practical-variants","runs":[{"text":"Two practical variants"}]},{"t":"p","runs":[{"text":"TruncatedSVD","b":true},{"text":" works on sparse matrices without centring them, which PCA cannot do. On TF-IDF matrices this is called latent semantic analysis, and it was the standard way to get dense document vectors for two decades before neural embeddings."}]},{"t":"p","runs":[{"text":"IncrementalPCA","b":true},{"text":" processes the data in batches, so you can fit it on more data than fits in memory."}]},{"t":"h2","id":"and-put-it-in-the-pipeline","runs":[{"text":"And put it in the pipeline"}]},{"t":"p","runs":[{"text":"PCA learns the components from data, so it must learn them from training rows only. Fitting PCA on the full dataset before splitting is on the leakage list in module 3 for exactly this reason, and it is one of the more commonly missed entries on that list."}]}],"check":{"question":[{"text":"A team applies PCA to 200 features, keeps components covering 95% of variance, and finds their classifier’s AUC drops from 0.86 to 0.79. What is the most likely explanation?"}],"options":[{"text":[{"text":"95% is too aggressive a threshold, and keeping 99% would always recover the lost performance."}],"correct":false,"tempting":[{"text":"Raising the threshold retains more components and may help, but the mechanism (that variance and predictiveness are different things) persists at any threshold."}]},{"text":[{"text":"PCA orders components by variance without reference to the target, so a discriminative direction carrying little variance can be among the 5% discarded."}],"correct":true,"tempting":null},{"text":[{"text":"PCA requires the target to be included in the matrix so the components align with the outcome."}],"correct":false,"tempting":[{"text":"Including the target would leak it into the features; PCA is unsupervised by design."}]},{"text":[{"text":"The components are orthogonal, which removes the feature interactions the classifier relied on."}],"correct":false,"tempting":[{"text":"Orthogonality removes correlation between components, but a model can still form interactions among them, so this is not what caused the loss."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"density-and-hierarchy","title":"DBSCAN and hierarchical clustering: when shape and structure matter","minutes":8},"next":{"slug":"seeing-high-dimensional-data","title":"t-SNE and UMAP, and how they mislead","minutes":8}}}