Support vector machines and the kernel trick

Maximising the gap

Many lines separate two classes. A support vector machine picks the one with the widest margin: the greatest distance to the nearest point of either class.

The intuition is that a boundary with room on both sides is more robust: a new point has to move further before it crosses. Only the points on the edge of the margin (the support vectors) determine the boundary. Every other training point could be deleted and the model would be identical, which is a unusual property and gives SVMs a compact representation.

Soft margins

Real data is not separable, so the SVM allows violations and charges for them. C sets the price:

• Small C: violations are cheap, so the margin is wide and some points sit inside or on the wrong side. More regularisation.
• Large C: violations are expensive, so the boundary contorts to classify every training point. Less regularisation, more overfitting.

C is the main hyperparameter and should be searched on a log scale: 0.01, 0.1, 1, 10, 100.

The kernel trick, which is not a trick

Here is the idea that made SVMs famous, and it is worth understanding properly because it recurs elsewhere.

Some data is not linearly separable in its original space but becomes separable in a higher-dimensional one. Points in a circle around the origin cannot be split by a line in two dimensions. Add a third coordinate equal to x^2 + y^2 and a flat plane separates them perfectly.

The obvious approach is to compute those extra coordinates. For a polynomial of degree 3 over 100 features that is over 170,000 columns, and for some useful transformations the space is infinite-dimensional.

The trick: the SVM’s optimisation only ever needs dot products between pairs of points, never the coordinates themselves. So if you can compute the dot product in the high-dimensional space directly from the original coordinates, you never need to visit that space.

RBF kernel: K(a, b) = exp(-gamma * ||a - b||^2)

That is a similarity between two points, computable in the original space, and it corresponds to a dot product in an infinite-dimensional feature space. You get the expressiveness of infinite dimensions for the cost of one exponential per pair.

Tuning gamma

With an RBF kernel, gamma controls how far a single point’s influence reaches.

• Small gamma: wide influence, smooth boundary, close to linear.
• Large gamma: each point influences only its immediate surroundings, and the boundary becomes islands around individual training points. This is overfitting, and it looks distinctive if you plot it.

C and gamma interact, so they must be searched together:

from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(StandardScaler(),
                      SVC(kernel="rbf", C=1.0, gamma="scale"))

gamma="scale" adapts to the data’s variance and is a sensible default. Scaling is mandatory: the RBF kernel is a function of Euclidean distance, so unscaled features break it exactly as they break k-NN.

The cost that decided their fate

Training an SVM is roughly quadratic to cubic in the number of rows, because the optimisation works with a matrix of pairwise similarities. At 10,000 rows this is fine. At 100,000 it is slow. At a million it is impractical.

That scaling is the main reason SVMs faded from general use after 2012, not any failure of the idea. Gradient boosting handles a million rows comfortably, and neural networks handle far more.

For a linear kernel there is a way out: LinearSVC uses a different solver that scales roughly linearly in rows, and it is fast on large sparse text data. If you want an SVM on a big dataset, that is the one.

Where they still earn their place

• Small to medium datasets with complex boundaries: under about 10,000 rows.
• High-dimensional data with few samples, such as gene expression with 20,000 features and 200 patients. SVMs are unusually well behaved here, where most methods struggle.
• Text classification with LinearSVC, which remains a strong and very fast baseline.
• When you need a hard decision boundary rather than a probability.

The honest weaknesses

No native probabilities: probability=True runs an internal cross-validated calibration that makes fitting several times slower. Two interacting hyperparameters that need a real search. Poor scaling in rows. And no interpretability at all with a non-linear kernel: the boundary lives in a space you never constructed and cannot inspect.

They are a beautiful piece of mathematics whose practical territory has narrowed considerably. Knowing them is still worthwhile, because the kernel idea (expressing a computation entirely in terms of similarities between pairs) appears again in Gaussian processes, in kernel PCA, and in the attention mechanism that transformers are built on.