Logistic regression, built from odds rather than assumed

Why not just fit a line to 0 and 1?

You can. It is called the linear probability model, statisticians use it deliberately in some settings, and for a beginner it fails in two visible ways.

Fit y = w*x + b where y is 0 or 1, and the line will happily predict 1.4 and -0.3. Those are not probabilities. Worse, the fit is dragged by the squared-error penalty on points that are already unambiguously classified: a customer the model puts at 0.99 who is a 1 still contributes error, and the line tilts to reduce it, damaging the boundary where the decision actually happens.

Odds, and then log-odds

Start from something a bookmaker would recognise. If a probability is 0.75, the odds are 0.75 / 0.25 = 3, said as three to one.

Probability lives in [0, 1]. Odds live in [0, infinity). Take the natural logarithm and log-odds live in (-infinity, +infinity), which is exactly the range a linear function produces.

So model the log-odds as linear:

log(p / (1 - p)) = w*x + b

Solve that for p and you get the logistic function, which people usually meet as an unexplained curve:

p = 1 / (1 + exp(-(w*x + b)))

The sigmoid was not chosen because it looks nice. It is what falls out when you decide to model log-odds with a straight line, and that decision was forced by needing an unbounded quantity for the linear part to produce.

Reading the numbers

The useful properties, worth committing to memory:

• Log-odds 0 means p = 0.5. Positive log-odds means over half.
• Log-odds 2.2 means p = 0.9. Log-odds -2.2 means p = 0.1.
• The curve is steepest at p = 0.5 and flat at the ends. Near the decision boundary, small changes in the inputs move the probability a lot; far from it, they barely move it at all.

That last property is why logistic regression is well behaved: it concentrates its attention where the decision is difficult, which is what the linear probability model failed to do.

The coefficient is an odds ratio

A coefficient of 0.7 on “has an overdraft” means: holding everything else fixed, having an overdraft adds 0.7 to the log-odds. Exponentiate and exp(0.7) = 2.01, so it roughly doubles the odds of the positive class.

Doubling the odds is not doubling the probability. From 0.10, doubled odds gives 0.18. From 0.40, it gives 0.57. From 0.80, it gives 0.89. The same coefficient produces a different change in probability depending on where you start, and this is the single most common misreading of a logistic model in a business meeting. The effect on probability is not constant; only the effect on log-odds is.

Fitting it

There is no closed form. The loss is log loss, from module 1, and it is convex (one bottom, no local minima) so an iterative optimiser reliably finds the global optimum. scikit-learn uses lbfgs by default, a quasi-Newton method that converges in far fewer iterations than plain gradient descent.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

clf = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))
clf.fit(X_tr, y_tr)
clf.predict_proba(X_te)[:, 1]        # probabilities, not labels

Use predict_proba, not predict. predict silently applies a threshold of 0.5, and 0.5 is almost never the right threshold. Module 5 is largely about that.

One trap worth naming: scikit-learn’s LogisticRegression applies L2 regularisation by default, controlled by C, where smaller C means stronger penalty. This surprises people coming from R or statsmodels, where the default is unpenalised. If you are comparing coefficients with a statistician’s output and they differ, this is usually why. Set penalty=None for an unpenalised fit.

Separation, and the coefficient that runs away

If some feature perfectly separates the classes (every row with is_fraud_flagged = 1 is fraud, no exceptions) then log loss keeps improving as that coefficient grows towards infinity, because pushing the predicted probability from 0.999 to 0.9999 still reduces the loss. Unpenalised, the optimiser chases it until it hits the iteration limit and reports a coefficient of 34.

This is called complete separation. It is a data problem masquerading as a convergence warning, and the default L2 penalty is what usually prevents you from ever noticing it. When you do see a coefficient in the tens, check whether that feature is a giveaway, and then check whether it is leakage, because a perfect separator in training data usually is.

What it is good at

Fast, calibrated out of the box in a way most models are not, gives one interpretable number per feature, and it is the standard in credit scoring and clinical prediction precisely because a regulator or a clinician can read it. When a gradient boosting model beats it by two points of AUC and cannot be explained to the person being refused a loan, the two points may not be worth it.