What a coefficient says, and the four ways it lies

The sentence a coefficient supports

Here is the only claim a linear coefficient licenses, and it is worth memorising in full:

Among rows like the ones in my training data, a one-unit increase in this feature is associated with a change of w in the target, among rows that are identical in every other feature in the model.

Every clause in that sentence is load-bearing, and each one corresponds to a way the reading goes wrong.

Lie one: unstandardised coefficients are not comparable

A coefficient of 0.03 on income and 4,500 on “number of children” does not mean children matter more. Income is measured in rupees and children in whole people. The magnitudes are units, not importance.

Fit on standardised features (mean 0, standard deviation 1) and the coefficients become comparable: each now says how much the target moves per one standard deviation of that feature. That is a meaningful comparison, and it is the only form in which “which feature matters most” can be read off a linear model at all.

from sklearn.preprocessing import StandardScaler
Z = StandardScaler().fit_transform(X)
# now coefficients are in target units per standard deviation

Lie two: correlated features split the credit arbitrarily

Put height_cm and height_inches into the same model. They carry the same information. The optimiser has infinitely many ways to divide the effect between them and will pick one for reasons that have nothing to do with the world: a rounding difference, the order of the columns, a random seed.

Refit on a slightly different sample and the split changes completely, sometimes flipping a sign. The predictions remain stable; the coefficients do not. This is multicollinearity and it is the reason coefficient-based feature importance is unreliable on real data, where features are correlated by default.

A quick diagnostic is the variance inflation factor, but there is a cheaper one: refit on five bootstrap samples and look at whether any coefficient changes sign. If it does, do not tell anyone what that feature means.

Lie three: the effect depends on what else is in the model

“Holding everything else fixed” is doing enormous work. Add a new column and existing coefficients move: sometimes a lot, sometimes reversing sign.

The cleanest illustration is Simpson’s paradox. Across a whole university, the acceptance rate for women may be lower than for men. Split by department and women may have the higher rate in every single department, because they applied disproportionately to the departments that reject most applicants. Both statements are arithmetically true. A model without a department column reports the first; a model with it reports the second. Neither is the coefficient “lying” in a technical sense; both are answering the question you asked, and the questions are different.

Which model is right depends on what you want to know, and that is a subject-matter question no algorithm can settle for you.

Lie four: statistical significance is not importance

With 5 million rows, effects of no practical consequence come out significant. With 300 rows, effects that matter enormously come out insignificant. A p-value is a statement about how surprising the data would be under a null hypothesis; it is not a measure of size and does not become one at any sample size.

Report the effect size and an interval. “Each extra year of tenure is associated with 1.2 fewer support tickets per year, 95% interval 0.4 to 2.0” is a sentence that supports a decision. “Tenure is significant at p < 0.001” is not.

Note that scikit-learn deliberately does not give you p-values. If you want them, statsmodels does, and its .summary() output is the standard table for this. It is free, and reading its output is a useful skill for anyone who will work near a statistician.

What coefficients are actually good for

Having spent four sections on limits, the positive case. Coefficients are excellent for:

• Sanity checking. A negative coefficient on income in a creditworthiness model is a bug alarm. Look at the sign of every coefficient you have a prior about; this catches more errors than any test.
• Finding leakage. One coefficient far larger than every other, on a feature you did not expect, is usually a column recorded after the outcome.
• Required explanations. Adverse action notices, in several jurisdictions, need a reason. A linear model can produce one per applicant, mechanically and consistently.
• Communication. “Every additional late payment adds about 12 points of risk” is a sentence that survives being repeated by someone who did not build the model. That has value that AUC does not.

Just write the sentence with its clauses intact, and refuse to let anyone drop them.