{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"logistic-regression-from-odds","title":"Logistic regression, built from odds rather than assumed","url":"https://addaly.com/learn/machine-learning-foundations/logistic-regression-from-odds","position":15,"of":92,"minutes":10,"answer":"Logistic regression models the log-odds as a straight line, which is why the sigmoid appears and why a coefficient multiplies the odds by a constant factor rather than shifting the probability by a constant amount.","keyIdea":"Logistic regression models the log-odds as a straight line, which is why the sigmoid appears and why a coefficient multiplies the odds by a constant factor rather than shifting the probability by a constant amount."},"body":{"format":"markdown","text":"## Why not just fit a line to 0 and 1?\n\nYou can. It is called the linear probability model, statisticians use it deliberately in some settings, and for a beginner it fails in two visible ways.\n\nFit `y = w*x + b` where `y` is 0 or 1, and the line will happily predict 1.4 and -0.3. Those are not probabilities. Worse, the fit is dragged by the squared-error penalty on points that are already unambiguously classified: a customer the model puts at 0.99 who is a 1 still contributes error, and the line tilts to reduce it, damaging the boundary where the decision actually happens.\n\n## Odds, and then log-odds\n\nStart from something a bookmaker would recognise. If a probability is 0.75, the odds are `0.75 / 0.25 = 3`, said as three to one.\n\nProbability lives in [0, 1]. Odds live in [0, infinity). Take the natural logarithm and log-odds live in (-infinity, +infinity), which is exactly the range a linear function produces.\n\nSo model the log-odds as linear:\n\n```text\nlog(p / (1 - p)) = w*x + b\n```\n\nSolve that for `p` and you get the logistic function, which people usually meet as an unexplained curve:\n\n```text\np = 1 / (1 + exp(-(w*x + b)))\n```\n\nThe sigmoid was not chosen because it looks nice. It is what falls out when you decide to model log-odds with a straight line, and that decision was forced by needing an unbounded quantity for the linear part to produce.\n\n## Reading the numbers\n\nThe useful properties, worth committing to memory:\n\n- Log-odds 0 means `p = 0.5`. Positive log-odds means over half.\n- Log-odds 2.2 means `p = 0.9`. Log-odds -2.2 means `p = 0.1`.\n- The curve is steepest at `p = 0.5` and flat at the ends. Near the decision boundary, small changes in the inputs move the probability a lot; far from it, they barely move it at all.\n\nThat last property is why logistic regression is well behaved: it concentrates its attention where the decision is difficult, which is what the linear probability model failed to do.\n\n## The coefficient is an odds ratio\n\nA coefficient of 0.7 on \"has an overdraft\" means: holding everything else fixed, having an overdraft adds 0.7 to the log-odds. Exponentiate and `exp(0.7) = 2.01`, so it roughly **doubles the odds** of the positive class.\n\nDoubling the odds is not doubling the probability. From 0.10, doubled odds gives 0.18. From 0.40, it gives 0.57. From 0.80, it gives 0.89. The same coefficient produces a different change in probability depending on where you start, and this is the single most common misreading of a logistic model in a business meeting. The effect on probability is not constant; only the effect on log-odds is.\n\n## Fitting it\n\nThere is no closed form. The loss is log loss, from module 1, and it is convex (one bottom, no local minima) so an iterative optimiser reliably finds the global optimum. scikit-learn uses `lbfgs` by default, a quasi-Newton method that converges in far fewer iterations than plain gradient descent.\n\n```python\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\n\nclf = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))\nclf.fit(X_tr, y_tr)\nclf.predict_proba(X_te)[:, 1]        # probabilities, not labels\n```\n\nUse `predict_proba`, not `predict`. `predict` silently applies a threshold of 0.5, and 0.5 is almost never the right threshold. Module 5 is largely about that.\n\nOne trap worth naming: scikit-learn's `LogisticRegression` applies **L2 regularisation by default**, controlled by `C`, where smaller `C` means stronger penalty. This surprises people coming from R or statsmodels, where the default is unpenalised. If you are comparing coefficients with a statistician's output and they differ, this is usually why. Set `penalty=None` for an unpenalised fit.\n\n## Separation, and the coefficient that runs away\n\nIf some feature perfectly separates the classes (every row with `is_fraud_flagged = 1` is fraud, no exceptions) then log loss keeps improving as that coefficient grows towards infinity, because pushing the predicted probability from 0.999 to 0.9999 still reduces the loss. Unpenalised, the optimiser chases it until it hits the iteration limit and reports a coefficient of 34.\n\nThis is called complete separation. It is a data problem masquerading as a convergence warning, and the default L2 penalty is what usually prevents you from ever noticing it. When you do see a coefficient in the tens, check whether that feature is a giveaway, and then check whether it is leakage, because a perfect separator in training data usually is.\n\n## What it is good at\n\nFast, calibrated out of the box in a way most models are not, gives one interpretable number per feature, and it is the standard in credit scoring and clinical prediction precisely because a regulator or a clinician can read it. When a gradient boosting model beats it by two points of AUC and cannot be explained to the person being refused a loan, the two points may not be worth it."},"check":{"question":"A model gives 'previous default' a coefficient of 0.69, so exp(0.69) is about 2. A manager concludes that previous default doubles a customer's chance of defaulting again. What is wrong with that reading?","options":[{"text":"The coefficient is on the log-odds scale, so it doubles the odds; the change in probability depends on the starting probability and is smaller than doubling except at very low base rates.","correct":true},{"text":"Coefficients from a regularised fit are shrunk toward zero, so the true effect is larger than 2.","correct":false,"tempting":"Correctly notes that L2 shrinks coefficients, but this concerns the size of the estimate, not the manager's conversion from log-odds to probability."},{"text":"The coefficient describes the average customer, so it cannot be applied to any individual.","correct":false,"tempting":"Sounds appropriately cautious but misstates the model: the coefficient does apply per row, just on the log-odds scale rather than the probability scale."},{"text":"Doubling only holds when all other features are zero, and they never are.","correct":false,"tempting":"Confuses the intercept's role with the coefficient's; the odds ratio holds while other features are held fixed at any values, not only at zero."}]},"prev":{"slug":"reading-a-loss-curve","title":"Diagnosing a model from the shape of its loss curve","url":"https://addaly.com/learn/machine-learning-foundations/reading-a-loss-curve"},"next":{"slug":"reading-coefficients","title":"What a coefficient says, and the four ways it lies","url":"https://addaly.com/learn/machine-learning-foundations/reading-coefficients"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Logistic regression, built from odds rather than assumed', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/logistic-regression-from-odds (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Logistic regression, built from odds rather than assumed","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"Logistic regression models the log-odds as a straight line, which is why the sigmoid appears and why a coefficient multiplies the odds by a constant factor rather than shifting the probability by a constant amount.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-not-just-fit-a-line-to-0-and-1","runs":[{"text":"Why not just fit a line to 0 and 1?"}]},{"t":"p","runs":[{"text":"You can. It is called the linear probability model, statisticians use it deliberately in some settings, and for a beginner it fails in two visible ways."}]},{"t":"p","runs":[{"text":"Fit "},{"text":"y = w*x + b","code":true},{"text":" where "},{"text":"y","code":true},{"text":" is 0 or 1, and the line will happily predict 1.4 and -0.3. Those are not probabilities. Worse, the fit is dragged by the squared-error penalty on points that are already unambiguously classified: a customer the model puts at 0.99 who is a 1 still contributes error, and the line tilts to reduce it, damaging the boundary where the decision actually happens."}]},{"t":"h2","id":"odds-and-then-log-odds","runs":[{"text":"Odds, and then log-odds"}]},{"t":"p","runs":[{"text":"Start from something a bookmaker would recognise. If a probability is 0.75, the odds are "},{"text":"0.75 / 0.25 = 3","code":true},{"text":", said as three to one."}]},{"t":"p","runs":[{"text":"Probability lives in [0, 1]. Odds live in [0, infinity). Take the natural logarithm and log-odds live in (-infinity, +infinity), which is exactly the range a linear function produces."}]},{"t":"p","runs":[{"text":"So model the log-odds as linear:"}]},{"t":"code","lang":"text","label":"Text","text":"log(p / (1 - p)) = w*x + b"},{"t":"p","runs":[{"text":"Solve that for "},{"text":"p","code":true},{"text":" and you get the logistic function, which people usually meet as an unexplained curve:"}]},{"t":"code","lang":"text","label":"Text","text":"p = 1 / (1 + exp(-(w*x + b)))"},{"t":"p","runs":[{"text":"The sigmoid was not chosen because it looks nice. It is what falls out when you decide to model log-odds with a straight line, and that decision was forced by needing an unbounded quantity for the linear part to produce."}]},{"t":"h2","id":"reading-the-numbers","runs":[{"text":"Reading the numbers"}]},{"t":"p","runs":[{"text":"The useful properties, worth committing to memory:"}]},{"t":"ul","items":[[{"text":"Log-odds 0 means "},{"text":"p = 0.5","code":true},{"text":". Positive log-odds means over half."}],[{"text":"Log-odds 2.2 means "},{"text":"p = 0.9","code":true},{"text":". Log-odds -2.2 means "},{"text":"p = 0.1","code":true},{"text":"."}],[{"text":"The curve is steepest at "},{"text":"p = 0.5","code":true},{"text":" and flat at the ends. Near the decision boundary, small changes in the inputs move the probability a lot; far from it, they barely move it at all."}]]},{"t":"p","runs":[{"text":"That last property is why logistic regression is well behaved: it concentrates its attention where the decision is difficult, which is what the linear probability model failed to do."}]},{"t":"h2","id":"the-coefficient-is-an-odds-ratio","runs":[{"text":"The coefficient is an odds ratio"}]},{"t":"p","runs":[{"text":"A coefficient of 0.7 on “has an overdraft” means: holding everything else fixed, having an overdraft adds 0.7 to the log-odds. Exponentiate and "},{"text":"exp(0.7) = 2.01","code":true},{"text":", so it roughly "},{"text":"doubles the odds","b":true},{"text":" of the positive class."}]},{"t":"p","runs":[{"text":"Doubling the odds is not doubling the probability. From 0.10, doubled odds gives 0.18. From 0.40, it gives 0.57. From 0.80, it gives 0.89. The same coefficient produces a different change in probability depending on where you start, and this is the single most common misreading of a logistic model in a business meeting. The effect on probability is not constant; only the effect on log-odds is."}]},{"t":"h2","id":"fitting-it","runs":[{"text":"Fitting it"}]},{"t":"p","runs":[{"text":"There is no closed form. The loss is log loss, from module 1, and it is convex (one bottom, no local minima) so an iterative optimiser reliably finds the global optimum. scikit-learn uses "},{"text":"lbfgs","code":true},{"text":" by default, a quasi-Newton method that converges in far fewer iterations than plain gradient descent."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\n\nclf = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))\nclf.fit(X_tr, y_tr)\nclf.predict_proba(X_te)[:, 1]        # probabilities, not labels"},{"t":"p","runs":[{"text":"Use "},{"text":"predict_proba","code":true},{"text":", not "},{"text":"predict","code":true},{"text":". "},{"text":"predict","code":true},{"text":" silently applies a threshold of 0.5, and 0.5 is almost never the right threshold. Module 5 is largely about that."}]},{"t":"p","runs":[{"text":"One trap worth naming: scikit-learn’s "},{"text":"LogisticRegression","code":true},{"text":" applies "},{"text":"L2 regularisation by default","b":true},{"text":", controlled by "},{"text":"C","code":true},{"text":", where smaller "},{"text":"C","code":true},{"text":" means stronger penalty. This surprises people coming from R or statsmodels, where the default is unpenalised. If you are comparing coefficients with a statistician’s output and they differ, this is usually why. Set "},{"text":"penalty=None","code":true},{"text":" for an unpenalised fit."}]},{"t":"h2","id":"separation-and-the-coefficient-that-runs-away","runs":[{"text":"Separation, and the coefficient that runs away"}]},{"t":"p","runs":[{"text":"If some feature perfectly separates the classes (every row with "},{"text":"is_fraud_flagged = 1","code":true},{"text":" is fraud, no exceptions) then log loss keeps improving as that coefficient grows towards infinity, because pushing the predicted probability from 0.999 to 0.9999 still reduces the loss. Unpenalised, the optimiser chases it until it hits the iteration limit and reports a coefficient of 34."}]},{"t":"p","runs":[{"text":"This is called complete separation. It is a data problem masquerading as a convergence warning, and the default L2 penalty is what usually prevents you from ever noticing it. When you do see a coefficient in the tens, check whether that feature is a giveaway, and then check whether it is leakage, because a perfect separator in training data usually is."}]},{"t":"h2","id":"what-it-is-good-at","runs":[{"text":"What it is good at"}]},{"t":"p","runs":[{"text":"Fast, calibrated out of the box in a way most models are not, gives one interpretable number per feature, and it is the standard in credit scoring and clinical prediction precisely because a regulator or a clinician can read it. When a gradient boosting model beats it by two points of AUC and cannot be explained to the person being refused a loan, the two points may not be worth it."}]}],"check":{"question":[{"text":"A model gives ‘previous default’ a coefficient of 0.69, so exp(0.69) is about 2. A manager concludes that previous default doubles a customer’s chance of defaulting again. What is wrong with that reading?"}],"options":[{"text":[{"text":"The coefficient is on the log-odds scale, so it doubles the odds; the change in probability depends on the starting probability and is smaller than doubling except at very low base rates."}],"correct":true,"tempting":null},{"text":[{"text":"Coefficients from a regularised fit are shrunk toward zero, so the true effect is larger than 2."}],"correct":false,"tempting":[{"text":"Correctly notes that L2 shrinks coefficients, but this concerns the size of the estimate, not the manager’s conversion from log-odds to probability."}]},{"text":[{"text":"The coefficient describes the average customer, so it cannot be applied to any individual."}],"correct":false,"tempting":[{"text":"Sounds appropriately cautious but misstates the model: the coefficient does apply per row, just on the log-odds scale rather than the probability scale."}]},{"text":[{"text":"Doubling only holds when all other features are zero, and they never are."}],"correct":false,"tempting":[{"text":"Confuses the intercept’s role with the coefficient’s; the odds ratio holds while other features are held fixed at any values, not only at zero."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"reading-a-loss-curve","title":"Diagnosing a model from the shape of its loss curve","minutes":8},"next":{"slug":"reading-coefficients","title":"What a coefficient says, and the four ways it lies","minutes":9}}}