{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"why-one-layer-is-not-enough","title":"XOR, and why the bend is the whole idea","url":"https://addaly.com/learn/machine-learning-foundations/why-one-layer-is-not-enough","position":74,"of":92,"minutes":8,"answer":"A stack of linear layers collapses algebraically into one linear layer, so every capability a network has beyond linear regression comes from the non-linearity applied between the layers.","keyIdea":"A stack of linear layers collapses algebraically into one linear layer, so every capability a network has beyond linear regression comes from the non-linearity applied between the layers."},"body":{"format":"markdown","text":"## Four points that broke the field for a decade\n\nFour inputs, one output:\n\n```text\n(0,0) -> 0\n(0,1) -> 1\n(1,0) -> 1\n(1,1) -> 0\n```\n\nThis is XOR: output 1 when exactly one input is 1. Try to separate the 1s from the 0s with a straight line. You cannot. The two positive points sit on one diagonal and the two negative points on the other, and no line puts them on opposite sides.\n\nA single-layer perceptron is a linear model with a step at the end, so it draws exactly one straight boundary. It cannot learn XOR. Minsky and Papert made this point in 1969, and interest in neural networks collapsed for most of the following decade.\n\n## The fix, and its catch\n\nAdd a hidden layer. Two units, each drawing its own line, and an output unit combining them. Unit A fires when at least one input is on; unit B fires when both are on; the output is A minus B, which is exactly XOR.\n\nThat works, but only because of something easy to miss. Suppose the hidden units are purely linear, with no bend applied to their output. Then:\n\n```text\nlayer 1: h = W1 x + b1\nlayer 2: y = W2 h + b2\n       = W2 (W1 x + b1) + b2\n       = (W2 W1) x + (W2 b1 + b2)\n```\n\nThat is `W x + b` for some other matrix `W`. **A stack of linear layers is a single linear layer.** Fifty layers, ten thousand units each, and it is still one linear model with a lot of wasted arithmetic. Depth buys nothing without a non-linearity between the layers.\n\nSo the essential component is not the layers. It is the bend applied after each one: ReLU, or sigmoid, or tanh. Everything a network can do that a linear model cannot comes from those bends.\n\n## What a network is, structurally\n\n```text\ninput -> [linear] -> [bend] -> [linear] -> [bend] -> [linear] -> output\n```\n\nEach linear step is a matrix multiplication plus a bias: the same operation as module 2's linear model, applied to many outputs at once. Each bend is a fixed function applied elementwise, with no parameters at all.\n\n```python\nimport numpy as np\ndef forward(x, W1, b1, W2, b2):\n    h = np.maximum(0, x @ W1 + b1)      # linear, then ReLU\n    return h @ W2 + b2                  # linear\n```\n\nThat is a complete two-layer network in three lines. Everything else in this module (better initialisation, better optimisers, normalisation, convolution, attention) is refinement of how those matrices are shaped and found.\n\n## What depth buys\n\nThe universal approximation theorem says a network with one hidden layer can approximate any continuous function on a bounded region to any accuracy, given enough units.\n\nRead the small print. It says such a network *exists*; it does not say gradient descent will find it, or that the number of units required is affordable. For many functions, a shallow network needs exponentially more units than a deep one to reach the same accuracy.\n\nThe intuitive reason is composition. Early layers learn simple pieces, later layers combine them into more complex ones, and a deep network can reuse the same piece in many combinations. A shallow one has to represent every combination separately. In vision this is visible directly: first-layer filters learn edges, later layers learn textures, then parts, then objects, and the edge detectors are shared by everything above.\n\n## Where the biological story helps and stops\n\nThe vocabulary (neurons, activation, firing) comes from a 1943 model of a biological neuron, and it is worth knowing that the analogy is loose and mostly historical.\n\nReal neurons communicate in spikes over time, have complex dendritic processing, and there is no known biological mechanism that computes backpropagation's exact gradients. Nothing in this module depends on the biology being accurate. A network is a differentiable function with many parameters, fitted by gradient descent, and that description is complete.\n\nThe useful mental model is the one from module 2 extended: a linear model whose features are themselves learned by earlier linear models with bends in between. The loss, the splits, the metrics and the thresholds all work exactly as before."},"check":{"question":"A team builds a 6-layer network for a tabular problem but omits the activation functions between layers. It performs identically to logistic regression. Why?","options":[{"text":"Composing linear maps yields another linear map, so the six layers reduce to a single equivalent linear layer regardless of their width.","correct":true},{"text":"Six layers is too few to show a benefit; with 20 layers the extra capacity would become visible.","correct":false,"tempting":"Depth cannot help while the composition remains linear, so no number of layers changes the outcome."},{"text":"Tabular data has no hierarchical structure, so depth cannot help on this kind of problem.","correct":false,"tempting":"Depth does tend to help less on tabular data, but that empirical tendency is not why this particular network is exactly equivalent to a linear model."},{"text":"The layers were not initialised properly, so the later ones never learned anything beyond the first.","correct":false,"tempting":"Poor initialisation produces slow or unstable training rather than an exact match to a linear model's performance."}]},"prev":{"slug":"neural-networks","title":"Neural networks as the natural next step","url":"https://addaly.com/learn/machine-learning-foundations/neural-networks"},"next":{"slug":"activations","title":"The bends: ReLU, and why sigmoid stopped being used inside networks","url":"https://addaly.com/learn/machine-learning-foundations/activations"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'XOR, and why the bend is the whole idea', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/why-one-layer-is-not-enough (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"XOR, and why the bend is the whole idea","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"A stack of linear layers collapses algebraically into one linear layer, so every capability a network has beyond linear regression comes from the non-linearity applied between the layers.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-points-that-broke-the-field-for-a-decade","runs":[{"text":"Four points that broke the field for a decade"}]},{"t":"p","runs":[{"text":"Four inputs, one output:"}]},{"t":"code","lang":"text","label":"Text","text":"(0,0) -> 0\n(0,1) -> 1\n(1,0) -> 1\n(1,1) -> 0"},{"t":"p","runs":[{"text":"This is XOR: output 1 when exactly one input is 1. Try to separate the 1s from the 0s with a straight line. You cannot. The two positive points sit on one diagonal and the two negative points on the other, and no line puts them on opposite sides."}]},{"t":"p","runs":[{"text":"A single-layer perceptron is a linear model with a step at the end, so it draws exactly one straight boundary. It cannot learn XOR. Minsky and Papert made this point in 1969, and interest in neural networks collapsed for most of the following decade."}]},{"t":"h2","id":"the-fix-and-its-catch","runs":[{"text":"The fix, and its catch"}]},{"t":"p","runs":[{"text":"Add a hidden layer. Two units, each drawing its own line, and an output unit combining them. Unit A fires when at least one input is on; unit B fires when both are on; the output is A minus B, which is exactly XOR."}]},{"t":"p","runs":[{"text":"That works, but only because of something easy to miss. Suppose the hidden units are purely linear, with no bend applied to their output. Then:"}]},{"t":"code","lang":"text","label":"Text","text":"layer 1: h = W1 x + b1\nlayer 2: y = W2 h + b2\n       = W2 (W1 x + b1) + b2\n       = (W2 W1) x + (W2 b1 + b2)"},{"t":"p","runs":[{"text":"That is "},{"text":"W x + b","code":true},{"text":" for some other matrix "},{"text":"W","code":true},{"text":". "},{"text":"A stack of linear layers is a single linear layer.","b":true},{"text":" Fifty layers, ten thousand units each, and it is still one linear model with a lot of wasted arithmetic. Depth buys nothing without a non-linearity between the layers."}]},{"t":"p","runs":[{"text":"So the essential component is not the layers. It is the bend applied after each one: ReLU, or sigmoid, or tanh. Everything a network can do that a linear model cannot comes from those bends."}]},{"t":"h2","id":"what-a-network-is-structurally","runs":[{"text":"What a network is, structurally"}]},{"t":"code","lang":"text","label":"Text","text":"input -> [linear] -> [bend] -> [linear] -> [bend] -> [linear] -> output"},{"t":"p","runs":[{"text":"Each linear step is a matrix multiplication plus a bias: the same operation as module 2’s linear model, applied to many outputs at once. Each bend is a fixed function applied elementwise, with no parameters at all."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\ndef forward(x, W1, b1, W2, b2):\n    h = np.maximum(0, x @ W1 + b1)      # linear, then ReLU\n    return h @ W2 + b2                  # linear"},{"t":"p","runs":[{"text":"That is a complete two-layer network in three lines. Everything else in this module (better initialisation, better optimisers, normalisation, convolution, attention) is refinement of how those matrices are shaped and found."}]},{"t":"h2","id":"what-depth-buys","runs":[{"text":"What depth buys"}]},{"t":"p","runs":[{"text":"The universal approximation theorem says a network with one hidden layer can approximate any continuous function on a bounded region to any accuracy, given enough units."}]},{"t":"p","runs":[{"text":"Read the small print. It says such a network "},{"text":"exists","i":true},{"text":"; it does not say gradient descent will find it, or that the number of units required is affordable. For many functions, a shallow network needs exponentially more units than a deep one to reach the same accuracy."}]},{"t":"p","runs":[{"text":"The intuitive reason is composition. Early layers learn simple pieces, later layers combine them into more complex ones, and a deep network can reuse the same piece in many combinations. A shallow one has to represent every combination separately. In vision this is visible directly: first-layer filters learn edges, later layers learn textures, then parts, then objects, and the edge detectors are shared by everything above."}]},{"t":"h2","id":"where-the-biological-story-helps-and-stops","runs":[{"text":"Where the biological story helps and stops"}]},{"t":"p","runs":[{"text":"The vocabulary (neurons, activation, firing) comes from a 1943 model of a biological neuron, and it is worth knowing that the analogy is loose and mostly historical."}]},{"t":"p","runs":[{"text":"Real neurons communicate in spikes over time, have complex dendritic processing, and there is no known biological mechanism that computes backpropagation’s exact gradients. Nothing in this module depends on the biology being accurate. A network is a differentiable function with many parameters, fitted by gradient descent, and that description is complete."}]},{"t":"p","runs":[{"text":"The useful mental model is the one from module 2 extended: a linear model whose features are themselves learned by earlier linear models with bends in between. The loss, the splits, the metrics and the thresholds all work exactly as before."}]}],"check":{"question":[{"text":"A team builds a 6-layer network for a tabular problem but omits the activation functions between layers. It performs identically to logistic regression. Why?"}],"options":[{"text":[{"text":"Composing linear maps yields another linear map, so the six layers reduce to a single equivalent linear layer regardless of their width."}],"correct":true,"tempting":null},{"text":[{"text":"Six layers is too few to show a benefit; with 20 layers the extra capacity would become visible."}],"correct":false,"tempting":[{"text":"Depth cannot help while the composition remains linear, so no number of layers changes the outcome."}]},{"text":[{"text":"Tabular data has no hierarchical structure, so depth cannot help on this kind of problem."}],"correct":false,"tempting":[{"text":"Depth does tend to help less on tabular data, but that empirical tendency is not why this particular network is exactly equivalent to a linear model."}]},{"text":[{"text":"The layers were not initialised properly, so the later ones never learned anything beyond the first."}],"correct":false,"tempting":[{"text":"Poor initialisation produces slow or unstable training rather than an exact match to a linear model’s performance."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"neural-networks","title":"Neural networks as the natural next step","minutes":9},"next":{"slug":"activations","title":"The bends: ReLU, and why sigmoid stopped being used inside networks","minutes":8}}}