{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"support-vector-machines","title":"Support vector machines and the kernel trick","url":"https://addaly.com/learn/machine-learning-foundations/support-vector-machines","position":60,"of":92,"minutes":9,"answer":"An SVM maximises the margin and depends only on the support vectors; the kernel trick works because the optimisation needs only pairwise dot products.","keyIdea":"An SVM maximises the margin and depends only on the support vectors; the kernel trick works because the optimisation needs only pairwise dot products, so an infinite-dimensional feature space costs one similarity computation instead of infinite coordinates."},"body":{"format":"markdown","text":"## Maximising the gap\n\nMany lines separate two classes. A support vector machine picks the one with the widest margin: the greatest distance to the nearest point of either class.\n\nThe intuition is that a boundary with room on both sides is more robust: a new point has to move further before it crosses. Only the points on the edge of the margin (the **support vectors**) determine the boundary. Every other training point could be deleted and the model would be identical, which is a unusual property and gives SVMs a compact representation.\n\n## Soft margins\n\nReal data is not separable, so the SVM allows violations and charges for them. `C` sets the price:\n\n- **Small C**: violations are cheap, so the margin is wide and some points sit inside or on the wrong side. More regularisation.\n- **Large C**: violations are expensive, so the boundary contorts to classify every training point. Less regularisation, more overfitting.\n\n`C` is the main hyperparameter and should be searched on a log scale: 0.01, 0.1, 1, 10, 100.\n\n## The kernel trick, which is not a trick\n\nHere is the idea that made SVMs famous, and it is worth understanding properly because it recurs elsewhere.\n\nSome data is not linearly separable in its original space but becomes separable in a higher-dimensional one. Points in a circle around the origin cannot be split by a line in two dimensions. Add a third coordinate equal to `x^2 + y^2` and a flat plane separates them perfectly.\n\nThe obvious approach is to compute those extra coordinates. For a polynomial of degree 3 over 100 features that is over 170,000 columns, and for some useful transformations the space is infinite-dimensional.\n\nThe trick: the SVM's optimisation only ever needs **dot products between pairs of points**, never the coordinates themselves. So if you can compute the dot product in the high-dimensional space directly from the original coordinates, you never need to visit that space.\n\n```text\nRBF kernel: K(a, b) = exp(-gamma * ||a - b||^2)\n```\n\nThat is a similarity between two points, computable in the original space, and it corresponds to a dot product in an infinite-dimensional feature space. You get the expressiveness of infinite dimensions for the cost of one exponential per pair.\n\n## Tuning gamma\n\nWith an RBF kernel, `gamma` controls how far a single point's influence reaches.\n\n- **Small gamma**: wide influence, smooth boundary, close to linear.\n- **Large gamma**: each point influences only its immediate surroundings, and the boundary becomes islands around individual training points. This is overfitting, and it looks distinctive if you plot it.\n\n`C` and `gamma` interact, so they must be searched together:\n\n```python\nfrom sklearn.svm import SVC\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nmodel = make_pipeline(StandardScaler(),\n                      SVC(kernel=\"rbf\", C=1.0, gamma=\"scale\"))\n```\n\n`gamma=\"scale\"` adapts to the data's variance and is a sensible default. Scaling is mandatory: the RBF kernel is a function of Euclidean distance, so unscaled features break it exactly as they break k-NN.\n\n## The cost that decided their fate\n\nTraining an SVM is roughly quadratic to cubic in the number of rows, because the optimisation works with a matrix of pairwise similarities. At 10,000 rows this is fine. At 100,000 it is slow. At a million it is impractical.\n\nThat scaling is the main reason SVMs faded from general use after 2012, not any failure of the idea. Gradient boosting handles a million rows comfortably, and neural networks handle far more.\n\nFor a linear kernel there is a way out: `LinearSVC` uses a different solver that scales roughly linearly in rows, and it is fast on large sparse text data. If you want an SVM on a big dataset, that is the one.\n\n## Where they still earn their place\n\n- **Small to medium datasets** with complex boundaries: under about 10,000 rows.\n- **High-dimensional data with few samples**, such as gene expression with 20,000 features and 200 patients. SVMs are unusually well behaved here, where most methods struggle.\n- **Text classification** with `LinearSVC`, which remains a strong and very fast baseline.\n- **When you need a hard decision boundary** rather than a probability.\n\n## The honest weaknesses\n\nNo native probabilities: `probability=True` runs an internal cross-validated calibration that makes fitting several times slower. Two interacting hyperparameters that need a real search. Poor scaling in rows. And no interpretability at all with a non-linear kernel: the boundary lives in a space you never constructed and cannot inspect.\n\nThey are a beautiful piece of mathematics whose practical territory has narrowed considerably. Knowing them is still worthwhile, because the kernel idea (expressing a computation entirely in terms of similarities between pairs) appears again in Gaussian processes, in kernel PCA, and in the attention mechanism that transformers are built on."},"check":{"question":"An RBF SVM achieves perfect training accuracy and 0.58 validation accuracy. Both C and gamma were set high. Which single change addresses the mechanism?","options":[{"text":"Switch to a polynomial kernel, which is less prone to memorising individual points.","correct":false,"tempting":"Changes the kernel family without addressing the settings that caused each point to dominate its own neighbourhood; a high-degree polynomial overfits similarly."},{"text":"Increase the number of training rows, since SVMs need large samples to find a stable margin.","correct":false,"tempting":"More data usually helps, but the configuration will continue to carve islands around points at any dataset size."},{"text":"Lower gamma, so each training point's influence extends further and the boundary becomes smooth instead of forming islands around individual points.","correct":true},{"text":"Enable probability=True so the decision boundary is fitted with calibrated probabilities rather than hard margins.","correct":false,"tempting":"Adds a post-hoc calibration step that changes the reported probabilities, not the shape of the fitted boundary."}]},"prev":{"slug":"naive-bayes","title":"Naive Bayes: wrong assumptions, useful results","url":"https://addaly.com/learn/machine-learning-foundations/naive-bayes"},"next":{"slug":"feature-importance-and-its-lies","title":"Feature importance, and the ways it misleads","url":"https://addaly.com/learn/machine-learning-foundations/feature-importance-and-its-lies"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Support vector machines and the kernel trick', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/support-vector-machines (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Support vector machines and the kernel trick","courseTitle":"Machine Learning, Foundations","module":{"index":6,"slug":"trees-and-ensembles","title":"Trees, ensembles, and the algorithms that win on tables"},"answer":null,"keyIdea":"An SVM maximises the margin and depends only on the support vectors; the kernel trick works because the optimisation needs only pairwise dot products, so an infinite-dimensional feature space costs one similarity computation instead of infinite coordinates.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"maximising-the-gap","runs":[{"text":"Maximising the gap"}]},{"t":"p","runs":[{"text":"Many lines separate two classes. A support vector machine picks the one with the widest margin: the greatest distance to the nearest point of either class."}]},{"t":"p","runs":[{"text":"The intuition is that a boundary with room on both sides is more robust: a new point has to move further before it crosses. Only the points on the edge of the margin (the "},{"text":"support vectors","b":true},{"text":") determine the boundary. Every other training point could be deleted and the model would be identical, which is a unusual property and gives SVMs a compact representation."}]},{"t":"h2","id":"soft-margins","runs":[{"text":"Soft margins"}]},{"t":"p","runs":[{"text":"Real data is not separable, so the SVM allows violations and charges for them. "},{"text":"C","code":true},{"text":" sets the price:"}]},{"t":"ul","items":[[{"text":"Small C","b":true},{"text":": violations are cheap, so the margin is wide and some points sit inside or on the wrong side. More regularisation."}],[{"text":"Large C","b":true},{"text":": violations are expensive, so the boundary contorts to classify every training point. Less regularisation, more overfitting."}]]},{"t":"p","runs":[{"text":"C","code":true},{"text":" is the main hyperparameter and should be searched on a log scale: 0.01, 0.1, 1, 10, 100."}]},{"t":"h2","id":"the-kernel-trick-which-is-not-a-trick","runs":[{"text":"The kernel trick, which is not a trick"}]},{"t":"p","runs":[{"text":"Here is the idea that made SVMs famous, and it is worth understanding properly because it recurs elsewhere."}]},{"t":"p","runs":[{"text":"Some data is not linearly separable in its original space but becomes separable in a higher-dimensional one. Points in a circle around the origin cannot be split by a line in two dimensions. Add a third coordinate equal to "},{"text":"x^2 + y^2","code":true},{"text":" and a flat plane separates them perfectly."}]},{"t":"p","runs":[{"text":"The obvious approach is to compute those extra coordinates. For a polynomial of degree 3 over 100 features that is over 170,000 columns, and for some useful transformations the space is infinite-dimensional."}]},{"t":"p","runs":[{"text":"The trick: the SVM’s optimisation only ever needs "},{"text":"dot products between pairs of points","b":true},{"text":", never the coordinates themselves. So if you can compute the dot product in the high-dimensional space directly from the original coordinates, you never need to visit that space."}]},{"t":"code","lang":"text","label":"Text","text":"RBF kernel: K(a, b) = exp(-gamma * ||a - b||^2)"},{"t":"p","runs":[{"text":"That is a similarity between two points, computable in the original space, and it corresponds to a dot product in an infinite-dimensional feature space. You get the expressiveness of infinite dimensions for the cost of one exponential per pair."}]},{"t":"h2","id":"tuning-gamma","runs":[{"text":"Tuning gamma"}]},{"t":"p","runs":[{"text":"With an RBF kernel, "},{"text":"gamma","code":true},{"text":" controls how far a single point’s influence reaches."}]},{"t":"ul","items":[[{"text":"Small gamma","b":true},{"text":": wide influence, smooth boundary, close to linear."}],[{"text":"Large gamma","b":true},{"text":": each point influences only its immediate surroundings, and the boundary becomes islands around individual training points. This is overfitting, and it looks distinctive if you plot it."}]]},{"t":"p","runs":[{"text":"C","code":true},{"text":" and "},{"text":"gamma","code":true},{"text":" interact, so they must be searched together:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.svm import SVC\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nmodel = make_pipeline(StandardScaler(),\n                      SVC(kernel=\"rbf\", C=1.0, gamma=\"scale\"))"},{"t":"p","runs":[{"text":"gamma=\"scale\"","code":true},{"text":" adapts to the data’s variance and is a sensible default. Scaling is mandatory: the RBF kernel is a function of Euclidean distance, so unscaled features break it exactly as they break k-NN."}]},{"t":"h2","id":"the-cost-that-decided-their-fate","runs":[{"text":"The cost that decided their fate"}]},{"t":"p","runs":[{"text":"Training an SVM is roughly quadratic to cubic in the number of rows, because the optimisation works with a matrix of pairwise similarities. At 10,000 rows this is fine. At 100,000 it is slow. At a million it is impractical."}]},{"t":"p","runs":[{"text":"That scaling is the main reason SVMs faded from general use after 2012, not any failure of the idea. Gradient boosting handles a million rows comfortably, and neural networks handle far more."}]},{"t":"p","runs":[{"text":"For a linear kernel there is a way out: "},{"text":"LinearSVC","code":true},{"text":" uses a different solver that scales roughly linearly in rows, and it is fast on large sparse text data. If you want an SVM on a big dataset, that is the one."}]},{"t":"h2","id":"where-they-still-earn-their-place","runs":[{"text":"Where they still earn their place"}]},{"t":"ul","items":[[{"text":"Small to medium datasets","b":true},{"text":" with complex boundaries: under about 10,000 rows."}],[{"text":"High-dimensional data with few samples","b":true},{"text":", such as gene expression with 20,000 features and 200 patients. SVMs are unusually well behaved here, where most methods struggle."}],[{"text":"Text classification","b":true},{"text":" with "},{"text":"LinearSVC","code":true},{"text":", which remains a strong and very fast baseline."}],[{"text":"When you need a hard decision boundary","b":true},{"text":" rather than a probability."}]]},{"t":"h2","id":"the-honest-weaknesses","runs":[{"text":"The honest weaknesses"}]},{"t":"p","runs":[{"text":"No native probabilities: "},{"text":"probability=True","code":true},{"text":" runs an internal cross-validated calibration that makes fitting several times slower. Two interacting hyperparameters that need a real search. Poor scaling in rows. And no interpretability at all with a non-linear kernel: the boundary lives in a space you never constructed and cannot inspect."}]},{"t":"p","runs":[{"text":"They are a beautiful piece of mathematics whose practical territory has narrowed considerably. Knowing them is still worthwhile, because the kernel idea (expressing a computation entirely in terms of similarities between pairs) appears again in Gaussian processes, in kernel PCA, and in the attention mechanism that transformers are built on."}]}],"check":{"question":[{"text":"An RBF SVM achieves perfect training accuracy and 0.58 validation accuracy. Both C and gamma were set high. Which single change addresses the mechanism?"}],"options":[{"text":[{"text":"Switch to a polynomial kernel, which is less prone to memorising individual points."}],"correct":false,"tempting":[{"text":"Changes the kernel family without addressing the settings that caused each point to dominate its own neighbourhood; a high-degree polynomial overfits similarly."}]},{"text":[{"text":"Increase the number of training rows, since SVMs need large samples to find a stable margin."}],"correct":false,"tempting":[{"text":"More data usually helps, but the configuration will continue to carve islands around points at any dataset size."}]},{"text":[{"text":"Lower gamma, so each training point’s influence extends further and the boundary becomes smooth instead of forming islands around individual points."}],"correct":true,"tempting":null},{"text":[{"text":"Enable probability=True so the decision boundary is fitted with calibrated probabilities rather than hard margins."}],"correct":false,"tempting":[{"text":"Adds a post-hoc calibration step that changes the reported probabilities, not the shape of the fitted boundary."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"naive-bayes","title":"Naive Bayes: wrong assumptions, useful results","minutes":8},"next":{"slug":"feature-importance-and-its-lies","title":"Feature importance, and the ways it misleads","minutes":9}}}