Machine Learning, Foundations · Chapter 08
The bend is the whole idea
1:55 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
Stack fifty linear layers with nothing between them and you have one linear layer with a great deal of wasted arithmetic, because matrix multiplication is associative and the whole thing collapses. Every capability a network has beyond a straight line comes from the bend applied between the layers. Early networks used the sigmoid, whose derivative is at most a quarter. Backpropagation multiplies those derivatives layer by layer, so ten layers down the gradient is about a millionth of what arrived at the top, and the early layers stop learning. A rectified linear unit is a hinge: negatives to zero, positives through unchanged. Its derivative is exactly one where it is active. That crude thing replaced the elegant one, and deep networks became trainable. You are not going to train a vision model from scratch, and you do not need to. Somebody already spent the compute and released the weights, and the early layers of that model learned edges and textures, which are the same structures whether the picture is a dog, a chest scan or a defective weld. So start at the cheapest level of adaptation. Freeze the whole pretrained model, use it to turn each image into a vector, and fit a logistic regression on those vectors. On a few hundred images that is often within a couple of points of full fine-tuning, and it runs on a laptop processor. Escalate only when the measurement says your data supports it.