{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"making-a-linear-model-curve","title":"Making a straight-line model bend","url":"https://addaly.com/learn/machine-learning-foundations/making-a-linear-model-curve","position":18,"of":92,"minutes":8,"answer":"A linear model bends by manufacturing columns (squares, interactions, bins, splines) and the interaction terms matter most, because finding those automatically is precisely what tree models do that linear models cannot.","keyIdea":"A linear model bends by manufacturing columns (squares, interactions, bins, splines) and the interaction terms matter most, because finding those automatically is precisely what tree models do that linear models cannot."},"body":{"format":"markdown","text":"## Linear in the parameters, not in the data\n\nThe word \"linear\" in linear regression refers to the parameters, not the features. `y = w1*x + w2*x^2 + b` is still a linear model: it is linear in `w1` and `w2`, which is what the normal equation and the convexity require. The fact that it draws a parabola is irrelevant to the fitting machinery.\n\nThis is more useful than it sounds, because it means you can give a linear model any shape you like by manufacturing columns.\n\n## Polynomial features\n\n```python\nfrom sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.linear_model import Ridge\n\nmodel = make_pipeline(\n    PolynomialFeatures(degree=2, include_bias=False),\n    StandardScaler(),\n    Ridge(alpha=1.0),\n)\n```\n\nWith three input features and degree 2, `PolynomialFeatures` produces `a, b, c, a^2, ab, ac, b^2, bc, c^2`: nine columns from three. That growth is the catch. At degree 3 with 20 features you get 1,770 columns, and the model has more freedom than data to constrain it. Degree 2 with a ridge penalty is a reasonable thing to try; degree 5 is almost never one.\n\nAnd polynomials misbehave at the edges. A degree-4 fit that looks sensible through the middle of your data will shoot off vertically just past the last point. If you take one habit from this lesson: plot the fitted curve over a range wider than your data before trusting it.\n\n## Interactions are usually the point\n\nThe cross terms matter more than the squares. `ab` lets the effect of `a` depend on the value of `b`, which is how most real relationships work.\n\nA discount raises conversion, but much more for price-sensitive customers than for corporate accounts. A linear model with `discount` and `is_corporate` as separate columns says the discount has one fixed effect for everyone. Add `discount * is_corporate` and it can say something different for each group. That single manufactured column is often worth more than a change of algorithm.\n\nThis is also the honest explanation of why gradient boosting beats linear models so often on tabular data. A tree finds interactions automatically by splitting on `b` and then on `a` inside that branch. A linear model finds them only if you thought of them and wrote the column. The gap is not that trees are cleverer; it is that they do this one thing without being asked.\n\n## Binning, and what it costs\n\nCut a continuous feature into ranges and one-hot encode them. Age becomes `18-25, 26-35, 36-50, 51+`, and now the model can fit any step-shaped relationship with no assumption about smoothness.\n\nThis is useful when the relationship has a real threshold: a policy that changes at 60, a tariff band, a legal age. It is wasteful otherwise: you have thrown away all the information inside each bin, so 26 and 35 become indistinguishable, and you have added a hard discontinuity at every boundary, so 35 and 36 differ sharply for no reason in the world.\n\nA rule that holds up: bin when the boundary is real, otherwise use a spline.\n\n## Splines: the option most people skip\n\nA spline fits separate low-degree polynomials on segments of the feature's range, joined so the result is smooth. You get flexible curvature without the wild edge behaviour of a high-degree polynomial, because no single polynomial has to cover the whole range.\n\n```python\nfrom sklearn.preprocessing import SplineTransformer\nmodel = make_pipeline(\n    SplineTransformer(n_knots=5, degree=3),\n    Ridge(alpha=1.0),\n)\n```\n\nFive knots and cubic segments is a good default. This is what a statistician reaches for when a relationship is clearly curved, and it is much better behaved than `PolynomialFeatures(degree=5)`. It is in scikit-learn, it is one line, and hardly anyone uses it.\n\n## Transform the target too\n\nIf the target is strongly right-skewed (income, house prices, time-to-event, almost any monetary quantity) fitting `log(y)` instead of `y` often does more than any feature transformation.\n\nIt makes the errors multiplicative rather than additive, which usually matches reality better: being wrong by ₹50,000 on a ₹2 crore property is a small error, and being wrong by ₹50,000 on a ₹3 lakh property is a disaster. Squared error on the raw target treats them identically. On the log scale it does not.\n\nTwo cautions. Use `np.log1p` if the target contains zeros. And remember that back-transforming with `np.exp` gives you the *median* of the predicted distribution, not the mean, so if you need an unbiased total across many rows, the naive exponential will underestimate it. `TransformedTargetRegressor` handles the mechanics; the bias is yours to think about.\n\n## When to stop\n\nIf you find yourself adding degree-4 polynomials, six interaction terms and manual bins, you have hand-built a worse gradient boosting model. Switch. The point of these techniques is to get a fast, explainable model closer to the truth, not to win a fight with a tree."},"check":{"question":"A conversion model has 'discount_pct' and 'is_enterprise' as separate features and fits poorly. A colleague suggests raising the polynomial degree to 3. What would more likely fix it, and why?","options":[{"text":"Adding the product discount_pct * is_enterprise, because the model currently forces one fixed discount effect on both customer types.","correct":true},{"text":"Binning discount_pct into deciles, because the relationship with conversion is probably non-monotonic.","correct":false,"tempting":"Binning adds flexibility within one feature, but the deficiency described is a between-feature dependence that binning alone cannot express."},{"text":"Raising the degree to 3 as suggested, since higher degree captures more complex relationships in general.","correct":false,"tempting":"Degree 3 does include the interaction, but it also adds many other terms and sharp edge behaviour, so it is a costly way to get the one term that is needed."},{"text":"Log-transforming discount_pct, because monetary and percentage features are usually skewed.","correct":false,"tempting":"Transforms the shape of one feature's effect, which does not let the effect differ between enterprise and non-enterprise customers."}]},"prev":{"slug":"ridge-and-lasso","title":"Regularisation: paying for large coefficients","url":"https://addaly.com/learn/machine-learning-foundations/ridge-and-lasso"},"next":{"slug":"more-than-two-classes","title":"More than two classes, and what changes","url":"https://addaly.com/learn/machine-learning-foundations/more-than-two-classes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Making a straight-line model bend', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/making-a-linear-model-curve (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Making a straight-line model bend","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"A linear model bends by manufacturing columns (squares, interactions, bins, splines) and the interaction terms matter most, because finding those automatically is precisely what tree models do that linear models cannot.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"linear-in-the-parameters-not-in-the-data","runs":[{"text":"Linear in the parameters, not in the data"}]},{"t":"p","runs":[{"text":"The word “linear” in linear regression refers to the parameters, not the features. "},{"text":"y = w1*x + w2*x^2 + b","code":true},{"text":" is still a linear model: it is linear in "},{"text":"w1","code":true},{"text":" and "},{"text":"w2","code":true},{"text":", which is what the normal equation and the convexity require. The fact that it draws a parabola is irrelevant to the fitting machinery."}]},{"t":"p","runs":[{"text":"This is more useful than it sounds, because it means you can give a linear model any shape you like by manufacturing columns."}]},{"t":"h2","id":"polynomial-features","runs":[{"text":"Polynomial features"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.linear_model import Ridge\n\nmodel = make_pipeline(\n    PolynomialFeatures(degree=2, include_bias=False),\n    StandardScaler(),\n    Ridge(alpha=1.0),\n)"},{"t":"p","runs":[{"text":"With three input features and degree 2, "},{"text":"PolynomialFeatures","code":true},{"text":" produces "},{"text":"a, b, c, a^2, ab, ac, b^2, bc, c^2","code":true},{"text":": nine columns from three. That growth is the catch. At degree 3 with 20 features you get 1,770 columns, and the model has more freedom than data to constrain it. Degree 2 with a ridge penalty is a reasonable thing to try; degree 5 is almost never one."}]},{"t":"p","runs":[{"text":"And polynomials misbehave at the edges. A degree-4 fit that looks sensible through the middle of your data will shoot off vertically just past the last point. If you take one habit from this lesson: plot the fitted curve over a range wider than your data before trusting it."}]},{"t":"h2","id":"interactions-are-usually-the-point","runs":[{"text":"Interactions are usually the point"}]},{"t":"p","runs":[{"text":"The cross terms matter more than the squares. "},{"text":"ab","code":true},{"text":" lets the effect of "},{"text":"a","code":true},{"text":" depend on the value of "},{"text":"b","code":true},{"text":", which is how most real relationships work."}]},{"t":"p","runs":[{"text":"A discount raises conversion, but much more for price-sensitive customers than for corporate accounts. A linear model with "},{"text":"discount","code":true},{"text":" and "},{"text":"is_corporate","code":true},{"text":" as separate columns says the discount has one fixed effect for everyone. Add "},{"text":"discount * is_corporate","code":true},{"text":" and it can say something different for each group. That single manufactured column is often worth more than a change of algorithm."}]},{"t":"p","runs":[{"text":"This is also the honest explanation of why gradient boosting beats linear models so often on tabular data. A tree finds interactions automatically by splitting on "},{"text":"b","code":true},{"text":" and then on "},{"text":"a","code":true},{"text":" inside that branch. A linear model finds them only if you thought of them and wrote the column. The gap is not that trees are cleverer; it is that they do this one thing without being asked."}]},{"t":"h2","id":"binning-and-what-it-costs","runs":[{"text":"Binning, and what it costs"}]},{"t":"p","runs":[{"text":"Cut a continuous feature into ranges and one-hot encode them. Age becomes "},{"text":"18-25, 26-35, 36-50, 51+","code":true},{"text":", and now the model can fit any step-shaped relationship with no assumption about smoothness."}]},{"t":"p","runs":[{"text":"This is useful when the relationship has a real threshold: a policy that changes at 60, a tariff band, a legal age. It is wasteful otherwise: you have thrown away all the information inside each bin, so 26 and 35 become indistinguishable, and you have added a hard discontinuity at every boundary, so 35 and 36 differ sharply for no reason in the world."}]},{"t":"p","runs":[{"text":"A rule that holds up: bin when the boundary is real, otherwise use a spline."}]},{"t":"h2","id":"splines-the-option-most-people-skip","runs":[{"text":"Splines: the option most people skip"}]},{"t":"p","runs":[{"text":"A spline fits separate low-degree polynomials on segments of the feature’s range, joined so the result is smooth. You get flexible curvature without the wild edge behaviour of a high-degree polynomial, because no single polynomial has to cover the whole range."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.preprocessing import SplineTransformer\nmodel = make_pipeline(\n    SplineTransformer(n_knots=5, degree=3),\n    Ridge(alpha=1.0),\n)"},{"t":"p","runs":[{"text":"Five knots and cubic segments is a good default. This is what a statistician reaches for when a relationship is clearly curved, and it is much better behaved than "},{"text":"PolynomialFeatures(degree=5)","code":true},{"text":". It is in scikit-learn, it is one line, and hardly anyone uses it."}]},{"t":"h2","id":"transform-the-target-too","runs":[{"text":"Transform the target too"}]},{"t":"p","runs":[{"text":"If the target is strongly right-skewed (income, house prices, time-to-event, almost any monetary quantity) fitting "},{"text":"log(y)","code":true},{"text":" instead of "},{"text":"y","code":true},{"text":" often does more than any feature transformation."}]},{"t":"p","runs":[{"text":"It makes the errors multiplicative rather than additive, which usually matches reality better: being wrong by ₹50,000 on a ₹2 crore property is a small error, and being wrong by ₹50,000 on a ₹3 lakh property is a disaster. Squared error on the raw target treats them identically. On the log scale it does not."}]},{"t":"p","runs":[{"text":"Two cautions. Use "},{"text":"np.log1p","code":true},{"text":" if the target contains zeros. And remember that back-transforming with "},{"text":"np.exp","code":true},{"text":" gives you the "},{"text":"median","i":true},{"text":" of the predicted distribution, not the mean, so if you need an unbiased total across many rows, the naive exponential will underestimate it. "},{"text":"TransformedTargetRegressor","code":true},{"text":" handles the mechanics; the bias is yours to think about."}]},{"t":"h2","id":"when-to-stop","runs":[{"text":"When to stop"}]},{"t":"p","runs":[{"text":"If you find yourself adding degree-4 polynomials, six interaction terms and manual bins, you have hand-built a worse gradient boosting model. Switch. The point of these techniques is to get a fast, explainable model closer to the truth, not to win a fight with a tree."}]}],"check":{"question":[{"text":"A conversion model has ‘discount_pct’ and ‘is_enterprise’ as separate features and fits poorly. A colleague suggests raising the polynomial degree to 3. What would more likely fix it, and why?"}],"options":[{"text":[{"text":"Adding the product discount_pct * is_enterprise, because the model currently forces one fixed discount effect on both customer types."}],"correct":true,"tempting":null},{"text":[{"text":"Binning discount_pct into deciles, because the relationship with conversion is probably non-monotonic."}],"correct":false,"tempting":[{"text":"Binning adds flexibility within one feature, but the deficiency described is a between-feature dependence that binning alone cannot express."}]},{"text":[{"text":"Raising the degree to 3 as suggested, since higher degree captures more complex relationships in general."}],"correct":false,"tempting":[{"text":"Degree 3 does include the interaction, but it also adds many other terms and sharp edge behaviour, so it is a costly way to get the one term that is needed."}]},{"text":[{"text":"Log-transforming discount_pct, because monetary and percentage features are usually skewed."}],"correct":false,"tempting":[{"text":"Transforms the shape of one feature’s effect, which does not let the effect differ between enterprise and non-enterprise customers."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"ridge-and-lasso","title":"Regularisation: paying for large coefficients","minutes":9},"next":{"slug":"more-than-two-classes","title":"More than two classes, and what changes","minutes":8}}}