Making a straight-line model bend

Linear in the parameters, not in the data

The word “linear” in linear regression refers to the parameters, not the features. y = w1*x + w2*x^2 + b is still a linear model: it is linear in w1 and w2, which is what the normal equation and the convexity require. The fact that it draws a parabola is irrelevant to the fitting machinery.

This is more useful than it sounds, because it means you can give a linear model any shape you like by manufacturing columns.

Polynomial features

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import Ridge

model = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    StandardScaler(),
    Ridge(alpha=1.0),
)

With three input features and degree 2, PolynomialFeatures produces a, b, c, a^2, ab, ac, b^2, bc, c^2: nine columns from three. That growth is the catch. At degree 3 with 20 features you get 1,770 columns, and the model has more freedom than data to constrain it. Degree 2 with a ridge penalty is a reasonable thing to try; degree 5 is almost never one.

And polynomials misbehave at the edges. A degree-4 fit that looks sensible through the middle of your data will shoot off vertically just past the last point. If you take one habit from this lesson: plot the fitted curve over a range wider than your data before trusting it.

Interactions are usually the point

The cross terms matter more than the squares. ab lets the effect of a depend on the value of b, which is how most real relationships work.

A discount raises conversion, but much more for price-sensitive customers than for corporate accounts. A linear model with discount and is_corporate as separate columns says the discount has one fixed effect for everyone. Add discount * is_corporate and it can say something different for each group. That single manufactured column is often worth more than a change of algorithm.

This is also the honest explanation of why gradient boosting beats linear models so often on tabular data. A tree finds interactions automatically by splitting on b and then on a inside that branch. A linear model finds them only if you thought of them and wrote the column. The gap is not that trees are cleverer; it is that they do this one thing without being asked.

Binning, and what it costs

Cut a continuous feature into ranges and one-hot encode them. Age becomes 18-25, 26-35, 36-50, 51+, and now the model can fit any step-shaped relationship with no assumption about smoothness.

This is useful when the relationship has a real threshold: a policy that changes at 60, a tariff band, a legal age. It is wasteful otherwise: you have thrown away all the information inside each bin, so 26 and 35 become indistinguishable, and you have added a hard discontinuity at every boundary, so 35 and 36 differ sharply for no reason in the world.

A rule that holds up: bin when the boundary is real, otherwise use a spline.

Splines: the option most people skip

A spline fits separate low-degree polynomials on segments of the feature’s range, joined so the result is smooth. You get flexible curvature without the wild edge behaviour of a high-degree polynomial, because no single polynomial has to cover the whole range.

from sklearn.preprocessing import SplineTransformer
model = make_pipeline(
    SplineTransformer(n_knots=5, degree=3),
    Ridge(alpha=1.0),
)

Five knots and cubic segments is a good default. This is what a statistician reaches for when a relationship is clearly curved, and it is much better behaved than PolynomialFeatures(degree=5). It is in scikit-learn, it is one line, and hardly anyone uses it.

Transform the target too

If the target is strongly right-skewed (income, house prices, time-to-event, almost any monetary quantity) fitting log(y) instead of y often does more than any feature transformation.

It makes the errors multiplicative rather than additive, which usually matches reality better: being wrong by ₹50,000 on a ₹2 crore property is a small error, and being wrong by ₹50,000 on a ₹3 lakh property is a disaster. Squared error on the raw target treats them identically. On the log scale it does not.

Two cautions. Use np.log1p if the target contains zeros. And remember that back-transforming with np.exp gives you the median of the predicted distribution, not the mean, so if you need an unbiased total across many rows, the naive exponential will underestimate it. TransformedTargetRegressor handles the mechanics; the bias is yours to think about.

When to stop

If you find yourself adding degree-4 polynomials, six interaction terms and manual bins, you have hand-built a worse gradient boosting model. Switch. The point of these techniques is to get a fast, explainable model closer to the truth, not to win a fight with a tree.