Fitting a line to five points, by hand

Five houses

Size in square metres, price in lakhs.

size   price
 50     32
 70     44
 90     51
110     67
130     74

We want price = w * size + b. Two numbers. Everything about model fitting is visible in finding them.

Guess, and measure how wrong you are

Start with w = 0.5, b = 0. Predictions are 25, 35, 45, 55, 65. The errors (actual minus predicted) are 7, 9, 6, 12, 9.

Mean squared error:

(49 + 81 + 36 + 144 + 81) / 5 = 391 / 5 = 78.2

Now try w = 0.55, b = 3. Predictions: 30.5, 41.5, 52.5, 63.5, 74.5. Errors: 1.5, 2.5, -1.5, 3.5, -0.5. MSE = (2.25 + 6.25 + 2.25 + 12.25 + 0.25) / 5 = 4.65.

The second model is better and we know it by a number, not by looking. That number is the entire mechanism. Everything else in this module is about finding good values of w and b faster than by guessing.

Which direction is downhill?

A useful thing about MSE is that you can compute, for the parameters you are currently holding, which way to move them. For one row the loss is (y - (w*x + b))^2, and the amount the loss changes when you nudge w is

d/dw = -2 * x * (y - prediction)

Read that without the calculus. It says: the push on w is proportional to the error, and scaled by x. A row where you are badly wrong pushes hard. A row with a large x pushes harder than one with a small x for the same error, which is exactly why unscaled features cause trouble later.

For the first house at w = 0.5, b = 0: error is 7, x is 50, so the derivative is -2 * 50 * 7 = -700. Negative, meaning increasing w reduces the loss. Averaged over all five rows, and moved a small step at a time, this is gradient descent, which you will meet properly in two lessons.

Doing it in one line

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[50], [70], [90], [110], [130]])
y = np.array([32, 44, 51, 67, 74])

m = LinearRegression().fit(X, y)
m.coef_, m.intercept_        # array([0.525]), 6.15
m.predict([[100]])           # array([58.65])

The best possible line for these five points is price = 0.525 * size + 6.15, MSE 3.585. Our hand guess of 0.55 and 3 got to 4.65, respectable, and it shows that near the optimum the loss surface is fairly flat, which is a property we will come back to.

What the two numbers mean

w = 0.525 means: within this data, each additional square metre is associated with 0.525 lakh more in price. Units are lakhs per square metre. A coefficient always has units, and stating them is the fastest way to catch a nonsense model.

b = 6.15 is the predicted price of a house of zero square metres, which does not exist. The intercept is usually not interpretable on its own; it is the number that positions the line vertically. If you centre your features first (subtract the mean from each) the intercept becomes the prediction at the average house, which is interpretable.

Two limitations you can see with five points

Extrapolation is a guess dressed as arithmetic. Ask this model for a 400 square metre house and it answers 216. It has never seen a house above 130 and has no way to know the relationship flattens, or that a house that large is a different market. The model will not warn you. Nothing in a linear model warns you when you leave the range of the training data, and this is the failure that most often reaches production quietly.

Five points cannot tell you the relationship is linear. They are consistent with a line. They are also consistent with a gentle curve, and with the true relationship being a step at 100 square metres. With five points you cannot distinguish these; with five hundred you often can. The straight line is an assumption you imposed, not a finding.

Try changing one number

Change the last price from 74 to 140, say a penthouse crept into the data, and refit. The coefficient jumps from 0.525 to about 0.855 and the whole line tilts. One row in five moved the answer by 63%.

That sensitivity is the squared-error property from the previous module, made visible. With 5,000 rows one outlier moves little. With 50 rows it moves a lot. It is worth knowing which regime you are in before you defend a coefficient in a meeting.

Why this simple thing survives

Linear regression is 220 years old and still the first model to fit on a new problem. It trains in milliseconds, gives a number per feature, extrapolates predictably rather than catastrophically, and needs no tuning. As the baseline lesson argued, you need something to beat. Half the time on tabular data, the fancy model beats this by less than anyone expected, and that fact is worth discovering in the first hour rather than the last.