XOR, and why the bend is the whole idea

Four points that broke the field for a decade

Four inputs, one output:

(0,0) -> 0
(0,1) -> 1
(1,0) -> 1
(1,1) -> 0

This is XOR: output 1 when exactly one input is 1. Try to separate the 1s from the 0s with a straight line. You cannot. The two positive points sit on one diagonal and the two negative points on the other, and no line puts them on opposite sides.

A single-layer perceptron is a linear model with a step at the end, so it draws exactly one straight boundary. It cannot learn XOR. Minsky and Papert made this point in 1969, and interest in neural networks collapsed for most of the following decade.

The fix, and its catch

Add a hidden layer. Two units, each drawing its own line, and an output unit combining them. Unit A fires when at least one input is on; unit B fires when both are on; the output is A minus B, which is exactly XOR.

That works, but only because of something easy to miss. Suppose the hidden units are purely linear, with no bend applied to their output. Then:

layer 1: h = W1 x + b1
layer 2: y = W2 h + b2
       = W2 (W1 x + b1) + b2
       = (W2 W1) x + (W2 b1 + b2)

That is W x + b for some other matrix W. A stack of linear layers is a single linear layer. Fifty layers, ten thousand units each, and it is still one linear model with a lot of wasted arithmetic. Depth buys nothing without a non-linearity between the layers.

So the essential component is not the layers. It is the bend applied after each one: ReLU, or sigmoid, or tanh. Everything a network can do that a linear model cannot comes from those bends.

What a network is, structurally

input -> [linear] -> [bend] -> [linear] -> [bend] -> [linear] -> output

Each linear step is a matrix multiplication plus a bias: the same operation as module 2’s linear model, applied to many outputs at once. Each bend is a fixed function applied elementwise, with no parameters at all.

import numpy as np
def forward(x, W1, b1, W2, b2):
    h = np.maximum(0, x @ W1 + b1)      # linear, then ReLU
    return h @ W2 + b2                  # linear

That is a complete two-layer network in three lines. Everything else in this module (better initialisation, better optimisers, normalisation, convolution, attention) is refinement of how those matrices are shaped and found.

What depth buys

The universal approximation theorem says a network with one hidden layer can approximate any continuous function on a bounded region to any accuracy, given enough units.

Read the small print. It says such a network exists; it does not say gradient descent will find it, or that the number of units required is affordable. For many functions, a shallow network needs exponentially more units than a deep one to reach the same accuracy.

The intuitive reason is composition. Early layers learn simple pieces, later layers combine them into more complex ones, and a deep network can reuse the same piece in many combinations. A shallow one has to represent every combination separately. In vision this is visible directly: first-layer filters learn edges, later layers learn textures, then parts, then objects, and the edge detectors are shared by everything above.

Where the biological story helps and stops

The vocabulary (neurons, activation, firing) comes from a 1943 model of a biological neuron, and it is worth knowing that the analogy is loose and mostly historical.

Real neurons communicate in spikes over time, have complex dendritic processing, and there is no known biological mechanism that computes backpropagation’s exact gradients. Nothing in this module depends on the biology being accurate. A network is a differentiable function with many parameters, fitted by gradient descent, and that description is complete.

The useful mental model is the one from module 2 extended: a linear model whose features are themselves learned by earlier linear models with bends in between. The loss, the splits, the metrics and the thresholds all work exactly as before.