Neural networks as the natural next step

Logistic regression, stacked

Logistic regression takes a weighted sum of the inputs and squashes it into a probability. Draw it and you get inputs on the left, one output on the right, a weight on every arrow. That is a neural network with one layer.

A network adds layers. The outputs of one layer become the inputs of the next:

h1 = relu(W1 @ x  + b1)   # 784 inputs -> 128 numbers
h2 = relu(W2 @ h1 + b2)   # 128 -> 64
y  = softmax(W3 @ h2 + b3)  # 64 -> 10 class probabilities

Three matrix multiplications, three additions, and a non-linear function between them. That is the whole computation.

Why the non-linearity is not optional

Remove the relu calls and the network is W3 @ (W2 @ (W1 @ x)). Matrix multiplication is associative, so that equals (W3 @ W2 @ W1) @ x, which is a single matrix. Three layers collapse to one. You can stack fifty of them and still have a linear model with a slower forward pass.

So something non-linear has to sit between the layers, and it can be very simple. relu(z) = max(0, z) — pass positives through, set negatives to zero. That one bend, repeated, is enough to let a deep stack express essentially any function.

The standard choices: ReLU in hidden layers, sigmoid on the output for binary classification, softmax on the output for multi-class.

What the layers are for

Lesson 3 said the features you hand over decide the ceiling. A network's hidden layers *are* learned features.

Train a network on satellite photos to find rooftop solar panels. Inspect what the layers respond to and you find, roughly: the first layer fires on edges and colour transitions; the middle layers on corners, repeating textures and rectangular grids; the last hidden layer on things that are recognisably panel arrays. Nobody designed those. They fell out of minimising the loss.

That is the real claim of deep learning, and it is narrower and more interesting than "bigger models are better". A network is not a superior curve-fitter. It is a system that does its own feature engineering, which is why it took over the domains where features were hardest to write by hand — vision, speech, language — and why it does not take over a spreadsheet with 18 columns you already understand.

Backpropagation is not a new idea

Gradient descent needs the gradient: how much the loss changes when each weight changes. In a network the weights sit several function-compositions away from the loss, so the derivatives compose too — that is the chain rule from calculus.

Backpropagation is the chain rule arranged efficiently. Run the input forward and keep the intermediate values. Then walk backward from the loss, and at each layer combine the gradient arriving from above with the values you saved, producing both the gradient for that layer's weights and the gradient to pass further back. One backward pass gives you the derivative with respect to every weight, at roughly the cost of one forward pass.

It is not a learning algorithm. It is how you get the number that Lesson 8's walk needs.

What carries over, and what gets harder

Everything in this course still applies, and most of it matters more.

• Loss, gradient descent, learning rate. Unchanged. The learning rate is still the hyperparameter that most often decides whether training works at all.
• Train, validation, test. Unchanged, and now enforced by people with millions of parameters and every incentive to peek.
• Overfitting. Now the default state. A network can memorise a training set outright, so regularization is not optional: dropout, weight decay, early stopping, data augmentation.
• Metrics and thresholds. Unchanged. A network's output is a score and the threshold is still yours.
• Features. Learned inside the model for perceptual data. Still yours for tabular data — which is why a gradient-boosted model will usually beat a network on 5,000 rows of a spreadsheet, and cost far less to train and run.

Where this goes

A large language model is the same three ingredients at a scale that changes the character of the thing. The shape is a transformer, whose layers pass information between positions in a sequence rather than only upward. The loss is: predict the next token. The optimiser is a refined gradient descent, running for weeks across thousands of accelerators.

The labels are free, because the next token is already in the text — the self-supervision from Lesson 2. That is what removed the bottleneck. Everything else you have learned here is still underneath it.