There is an exact answer, and we usually refuse it

The formula that ends the search

For linear regression with squared error, you do not need to search at all. There is an exact solution, written in one line of linear algebra:

w = (X'X)^-1 X'y

where X' is the transpose and ^-1 is the matrix inverse. This is the normal equation, and it gives the parameters that minimise MSE exactly, in one step, with no learning rate, no iterations and no convergence to worry about.

import numpy as np
X = np.array([[50], [70], [90], [110], [130]])
X1 = np.hstack([np.ones((5, 1)), X])          # add the intercept column
y = np.array([32, 44, 51, 67, 74])
np.linalg.solve(X1.T @ X1, X1.T @ y)          # array([6.15, 0.525])

Exactly the numbers scikit-learn returned. In fact LinearRegression does something very close to this internally: it uses a numerically safer routine called least squares via SVD, but the spirit is the same. There is no gradient descent inside it.

So why does every course teach gradient descent?

Three reasons the exact answer runs out

1. Cost. X'X is a p by p matrix where p is the number of features, and inverting it costs roughly p^3 operations. At 10 features that is nothing. At 1,000 features it is a billion operations, still fine. At 100,000 features, which a text model with one column per word reaches easily, it is 10^15 operations and a matrix of 10 billion entries. The formula has not become wrong; it has become unusable.

2. Memory. The closed form needs all of X at once. Gradient descent needs one batch at a time, which is why models train on datasets larger than the machine’s RAM. This is not a minor engineering convenience: it is the reason large models are trainable at all.

3. Most losses have no closed form. This is the deep reason. The normal equation exists because squared error against a linear function is a quadratic bowl with one bottom that calculus can locate. Change the loss to log loss, or the model to anything with a nonlinearity in it, and no such formula exists. Logistic regression (next lesson but one) has no closed form. Neural networks certainly do not. Gradient descent is the general method; the normal equation is a lucky special case.

The singular matrix, and what it is telling you

Sometimes X'X cannot be inverted at all. The mathematical term is singular; the practical meaning is that two or more of your columns carry the same information.

The classic way to cause it: one-hot encode a categorical column into all of its levels and keep the intercept. If is_mumbai + is_delhi + is_chennai = 1 for every row, and the intercept column is also all 1s, then one column is an exact combination of others. There are infinitely many parameter sets that fit equally well, so there is no unique answer, and the inverse does not exist.

Numerically it usually does not crash. It produces enormous coefficients of opposite sign that cancel (+184,000 on one city and -183,000 on another) which is the signature you should learn to recognise. When you see coefficients that large on ordinary data, look for collinear columns before you look for anything else.

The fixes: drop one level of each categorical (drop='first' in scikit-learn’s OneHotEncoder), or add a ridge penalty, which we cover later in this module and which makes the matrix invertible by construction.

Which one runs when you call .fit()

A useful piece of practical knowledge, because the library names are opaque:

• LinearRegression: least squares via SVD. Exact, no iterations, no learning rate. Good up to tens of thousands of features.
• Ridge with solver='auto': closed form with the penalty added, usually.
• SGDRegressor and SGDClassifier: iterative. Use these when the data does not fit in memory or when you want to fit incrementally with partial_fit.
• LogisticRegression: always iterative. There is no alternative; the max_iter warning you have probably seen is this optimiser saying it stopped before converging.

That last point explains a common confusion. “Increase max_iter” is the standard advice for the logistic regression convergence warning, and it works, but the more effective fix is usually to scale your features: an optimiser on badly scaled data takes many more steps to reach the same place, for reasons the next two lessons make concrete.

The summary

Exact solutions are better when they exist and are affordable. They exist for one loss and one model family. The field standardised on iterative optimisation not because it is more elegant but because it is the only method that survives contact with a nonlinear model, and once you have built the machinery you use it everywhere.