{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"reading-coefficients","title":"What a coefficient says, and the four ways it lies","url":"https://addaly.com/learn/machine-learning-foundations/reading-coefficients","position":16,"of":92,"minutes":9,"answer":"A coefficient is a change in the target per unit of one feature, holding the others in the model fixed, so it is only comparable when standardised, only stable when features are uncorrelated.","keyIdea":"A coefficient is a change in the target per unit of one feature, holding the others in the model fixed, so it is only comparable when standardised, only stable when features are uncorrelated, and only meaningful relative to the exact set of columns included."},"body":{"format":"markdown","text":"## The sentence a coefficient supports\n\nHere is the only claim a linear coefficient licenses, and it is worth memorising in full:\n\n> Among rows like the ones in my training data, a one-unit increase in this feature is associated with a change of `w` in the target, **among rows that are identical in every other feature in the model**.\n\nEvery clause in that sentence is load-bearing, and each one corresponds to a way the reading goes wrong.\n\n## Lie one: unstandardised coefficients are not comparable\n\nA coefficient of 0.03 on income and 4,500 on \"number of children\" does not mean children matter more. Income is measured in rupees and children in whole people. The magnitudes are units, not importance.\n\nFit on standardised features (mean 0, standard deviation 1) and the coefficients become comparable: each now says how much the target moves per one standard deviation of that feature. That is a meaningful comparison, and it is the only form in which \"which feature matters most\" can be read off a linear model at all.\n\n```python\nfrom sklearn.preprocessing import StandardScaler\nZ = StandardScaler().fit_transform(X)\n# now coefficients are in target units per standard deviation\n```\n\n## Lie two: correlated features split the credit arbitrarily\n\nPut `height_cm` and `height_inches` into the same model. They carry the same information. The optimiser has infinitely many ways to divide the effect between them and will pick one for reasons that have nothing to do with the world: a rounding difference, the order of the columns, a random seed.\n\nRefit on a slightly different sample and the split changes completely, sometimes flipping a sign. The *predictions* remain stable; the *coefficients* do not. This is multicollinearity and it is the reason coefficient-based feature importance is unreliable on real data, where features are correlated by default.\n\nA quick diagnostic is the variance inflation factor, but there is a cheaper one: refit on five bootstrap samples and look at whether any coefficient changes sign. If it does, do not tell anyone what that feature means.\n\n## Lie three: the effect depends on what else is in the model\n\n\"Holding everything else fixed\" is doing enormous work. Add a new column and existing coefficients move: sometimes a lot, sometimes reversing sign.\n\nThe cleanest illustration is Simpson's paradox. Across a whole university, the acceptance rate for women may be lower than for men. Split by department and women may have the higher rate in every single department, because they applied disproportionately to the departments that reject most applicants. Both statements are arithmetically true. A model without a department column reports the first; a model with it reports the second. Neither is the coefficient \"lying\" in a technical sense; both are answering the question you asked, and the questions are different.\n\nWhich model is right depends on what you want to know, and that is a subject-matter question no algorithm can settle for you.\n\n## Lie four: statistical significance is not importance\n\nWith 5 million rows, effects of no practical consequence come out significant. With 300 rows, effects that matter enormously come out insignificant. A p-value is a statement about how surprising the data would be under a null hypothesis; it is not a measure of size and does not become one at any sample size.\n\nReport the effect size and an interval. \"Each extra year of tenure is associated with 1.2 fewer support tickets per year, 95% interval 0.4 to 2.0\" is a sentence that supports a decision. \"Tenure is significant at p < 0.001\" is not.\n\nNote that scikit-learn deliberately does not give you p-values. If you want them, `statsmodels` does, and its `.summary()` output is the standard table for this. It is free, and reading its output is a useful skill for anyone who will work near a statistician.\n\n## What coefficients are actually good for\n\nHaving spent four sections on limits, the positive case. Coefficients are excellent for:\n\n- **Sanity checking.** A negative coefficient on income in a creditworthiness model is a bug alarm. Look at the sign of every coefficient you have a prior about; this catches more errors than any test.\n- **Finding leakage.** One coefficient far larger than every other, on a feature you did not expect, is usually a column recorded after the outcome.\n- **Required explanations.** Adverse action notices, in several jurisdictions, need a reason. A linear model can produce one per applicant, mechanically and consistently.\n- **Communication.** \"Every additional late payment adds about 12 points of risk\" is a sentence that survives being repeated by someone who did not build the model. That has value that AUC does not.\n\nJust write the sentence with its clauses intact, and refuse to let anyone drop them."},"check":{"question":"A team adds a 'region' feature to a pricing model and the coefficient on 'floor area' drops by 40%. They ask which version is correct. What is the right response?","options":[{"text":"The second, because more features always means less omitted-variable bias and a more accurate coefficient.","correct":false,"tempting":"Treats adding controls as monotonically improving, which is not true: controlling for a variable downstream of the effect can introduce bias rather than remove it."},{"text":"The first, because adding a correlated feature has split the credit and understated floor area's true effect.","correct":false,"tempting":"Assumes the smaller coefficient must be the distorted one, which is not determinable from the drop alone."},{"text":"Neither is 'correct' in isolation: the coefficients answer different questions (with and without holding region fixed) and which you want depends on whether region is a confounder or part of the effect you are trying to measure.","correct":true},{"text":"Whichever model has the better validation score, since predictive accuracy settles which coefficient set is more trustworthy.","correct":false,"tempting":"Uses predictive fit to adjudicate an interpretive question; two models can predict equally well while attributing the effect completely differently."}]},"prev":{"slug":"logistic-regression-from-odds","title":"Logistic regression, built from odds rather than assumed","url":"https://addaly.com/learn/machine-learning-foundations/logistic-regression-from-odds"},"next":{"slug":"ridge-and-lasso","title":"Regularisation: paying for large coefficients","url":"https://addaly.com/learn/machine-learning-foundations/ridge-and-lasso"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a coefficient says, and the four ways it lies', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/reading-coefficients (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a coefficient says, and the four ways it lies","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"A coefficient is a change in the target per unit of one feature, holding the others in the model fixed, so it is only comparable when standardised, only stable when features are uncorrelated, and only meaningful relative to the exact set of columns included.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-sentence-a-coefficient-supports","runs":[{"text":"The sentence a coefficient supports"}]},{"t":"p","runs":[{"text":"Here is the only claim a linear coefficient licenses, and it is worth memorising in full:"}]},{"t":"quote","runs":[{"text":"Among rows like the ones in my training data, a one-unit increase in this feature is associated with a change of "},{"text":"w","code":true},{"text":" in the target, "},{"text":"among rows that are identical in every other feature in the model","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"Every clause in that sentence is load-bearing, and each one corresponds to a way the reading goes wrong."}]},{"t":"h2","id":"lie-one-unstandardised-coefficients-are-not-comparable","runs":[{"text":"Lie one: unstandardised coefficients are not comparable"}]},{"t":"p","runs":[{"text":"A coefficient of 0.03 on income and 4,500 on “number of children” does not mean children matter more. Income is measured in rupees and children in whole people. The magnitudes are units, not importance."}]},{"t":"p","runs":[{"text":"Fit on standardised features (mean 0, standard deviation 1) and the coefficients become comparable: each now says how much the target moves per one standard deviation of that feature. That is a meaningful comparison, and it is the only form in which “which feature matters most” can be read off a linear model at all."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.preprocessing import StandardScaler\nZ = StandardScaler().fit_transform(X)\n# now coefficients are in target units per standard deviation"},{"t":"h2","id":"lie-two-correlated-features-split-the-credit-arbitrarily","runs":[{"text":"Lie two: correlated features split the credit arbitrarily"}]},{"t":"p","runs":[{"text":"Put "},{"text":"height_cm","code":true},{"text":" and "},{"text":"height_inches","code":true},{"text":" into the same model. They carry the same information. The optimiser has infinitely many ways to divide the effect between them and will pick one for reasons that have nothing to do with the world: a rounding difference, the order of the columns, a random seed."}]},{"t":"p","runs":[{"text":"Refit on a slightly different sample and the split changes completely, sometimes flipping a sign. The "},{"text":"predictions","i":true},{"text":" remain stable; the "},{"text":"coefficients","i":true},{"text":" do not. This is multicollinearity and it is the reason coefficient-based feature importance is unreliable on real data, where features are correlated by default."}]},{"t":"p","runs":[{"text":"A quick diagnostic is the variance inflation factor, but there is a cheaper one: refit on five bootstrap samples and look at whether any coefficient changes sign. If it does, do not tell anyone what that feature means."}]},{"t":"h2","id":"lie-three-the-effect-depends-on-what-else-is-in-the-model","runs":[{"text":"Lie three: the effect depends on what else is in the model"}]},{"t":"p","runs":[{"text":"“Holding everything else fixed” is doing enormous work. Add a new column and existing coefficients move: sometimes a lot, sometimes reversing sign."}]},{"t":"p","runs":[{"text":"The cleanest illustration is Simpson’s paradox. Across a whole university, the acceptance rate for women may be lower than for men. Split by department and women may have the higher rate in every single department, because they applied disproportionately to the departments that reject most applicants. Both statements are arithmetically true. A model without a department column reports the first; a model with it reports the second. Neither is the coefficient “lying” in a technical sense; both are answering the question you asked, and the questions are different."}]},{"t":"p","runs":[{"text":"Which model is right depends on what you want to know, and that is a subject-matter question no algorithm can settle for you."}]},{"t":"h2","id":"lie-four-statistical-significance-is-not-importance","runs":[{"text":"Lie four: statistical significance is not importance"}]},{"t":"p","runs":[{"text":"With 5 million rows, effects of no practical consequence come out significant. With 300 rows, effects that matter enormously come out insignificant. A p-value is a statement about how surprising the data would be under a null hypothesis; it is not a measure of size and does not become one at any sample size."}]},{"t":"p","runs":[{"text":"Report the effect size and an interval. “Each extra year of tenure is associated with 1.2 fewer support tickets per year, 95% interval 0.4 to 2.0” is a sentence that supports a decision. “Tenure is significant at p < 0.001” is not."}]},{"t":"p","runs":[{"text":"Note that scikit-learn deliberately does not give you p-values. If you want them, "},{"text":"statsmodels","code":true},{"text":" does, and its "},{"text":".summary()","code":true},{"text":" output is the standard table for this. It is free, and reading its output is a useful skill for anyone who will work near a statistician."}]},{"t":"h2","id":"what-coefficients-are-actually-good-for","runs":[{"text":"What coefficients are actually good for"}]},{"t":"p","runs":[{"text":"Having spent four sections on limits, the positive case. Coefficients are excellent for:"}]},{"t":"ul","items":[[{"text":"Sanity checking.","b":true},{"text":" A negative coefficient on income in a creditworthiness model is a bug alarm. Look at the sign of every coefficient you have a prior about; this catches more errors than any test."}],[{"text":"Finding leakage.","b":true},{"text":" One coefficient far larger than every other, on a feature you did not expect, is usually a column recorded after the outcome."}],[{"text":"Required explanations.","b":true},{"text":" Adverse action notices, in several jurisdictions, need a reason. A linear model can produce one per applicant, mechanically and consistently."}],[{"text":"Communication.","b":true},{"text":" “Every additional late payment adds about 12 points of risk” is a sentence that survives being repeated by someone who did not build the model. That has value that AUC does not."}]]},{"t":"p","runs":[{"text":"Just write the sentence with its clauses intact, and refuse to let anyone drop them."}]}],"check":{"question":[{"text":"A team adds a ‘region’ feature to a pricing model and the coefficient on ‘floor area’ drops by 40%. They ask which version is correct. What is the right response?"}],"options":[{"text":[{"text":"The second, because more features always means less omitted-variable bias and a more accurate coefficient."}],"correct":false,"tempting":[{"text":"Treats adding controls as monotonically improving, which is not true: controlling for a variable downstream of the effect can introduce bias rather than remove it."}]},{"text":[{"text":"The first, because adding a correlated feature has split the credit and understated floor area’s true effect."}],"correct":false,"tempting":[{"text":"Assumes the smaller coefficient must be the distorted one, which is not determinable from the drop alone."}]},{"text":[{"text":"Neither is ‘correct’ in isolation: the coefficients answer different questions (with and without holding region fixed) and which you want depends on whether region is a confounder or part of the effect you are trying to measure."}],"correct":true,"tempting":null},{"text":[{"text":"Whichever model has the better validation score, since predictive accuracy settles which coefficient set is more trustworthy."}],"correct":false,"tempting":[{"text":"Uses predictive fit to adjudicate an interpretive question; two models can predict equally well while attributing the effect completely differently."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"logistic-regression-from-odds","title":"Logistic regression, built from odds rather than assumed","minutes":10},"next":{"slug":"ridge-and-lasso","title":"Regularisation: paying for large coefficients","minutes":9}}}