Convolution: the same detector, everywhere in the image

Why a fully connected layer fails on images

A 224 by 224 colour image is 150,528 numbers. A fully connected layer with 1,000 units needs 150 million weights for the first layer alone. That is unaffordable, and worse, it is wasteful in a specific way: a cat detector learned for the top-left corner would have to be learned again, from separate examples, for every other position.

The fully connected layer knows nothing about the fact that neighbouring pixels are related and that a pattern means the same thing wherever it appears.

What a convolution does instead

Slide a small window (typically 3 by 3) across the image. At each position, multiply the window’s values by a small set of learned weights and sum. That set of weights is a filter or kernel.

nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)

This layer has 64 filters, each 3 by 3 across 3 input channels: 64 * 3 * 3 * 3 + 64 = 1,792 parameters. Compare that with 150 million.

Two ideas make the saving possible.

Local connectivity. Each output depends on a 3 by 3 patch, not the whole image. Pixels far apart are combined only in later layers, and only after the nearby structure has been summarised.

Weight sharing. The same filter is applied at every position. Learn an edge detector once and it works everywhere. This is what gives convolutional networks their translation equivariance: move the object in the image and the response moves with it.

What the filters learn

The learned filters are inspectable, and what they contain is consistent across networks and datasets.

• Layer 1: edges at various orientations, colour blobs. These look strikingly like Gabor filters, which is what early visual cortex is understood to compute: a convergence nobody designed.
• Layer 2 to 3: corners, curves, simple textures.
• Middle layers: repeating patterns, object parts: a wheel, an eye, a doorway.
• Late layers: whole objects and scene types.

Each layer combines the previous layer’s pieces. This is the compositional hierarchy from the depth lesson, made visible, and it is the reason transfer learning works so well in vision: the early layers are learning something general about images rather than something specific to the training task.

The receptive field

One 3 by 3 filter sees 3 pixels. Stack two and the second sees a 5 by 5 region of the original. Stack three and it is 7 by 7. Add pooling or a stride and the region grows much faster.

This is why depth matters in vision. A network needs a receptive field covering enough of the image to recognise a whole object, and stacking small filters is a cheaper way to get there than using one large one: two 3 by 3 layers have 18 weights per channel pair and cover the same region as a single 5 by 5 with 25, while including an extra non-linearity.

Pooling, and its decline

Max pooling takes the maximum over each 2 by 2 block, halving the spatial dimensions. It reduces computation and gives a little tolerance to small shifts.

Modern architectures increasingly use strided convolutions instead (a convolution that moves 2 pixels at a time) which achieves the same downsampling with learned weights rather than a fixed rule. Pooling is not gone, and it is no longer automatic.

Where the field went

Two developments worth knowing so the picture is current.

Depthwise separable convolutions split a convolution into a per-channel spatial step and a 1 by 1 mixing step, cutting the parameter count by roughly a factor of 8 to 9 with little accuracy loss. This is what makes MobileNet and its relatives run on a phone, and it is the reason on-device vision is practical at all.

Vision transformers cut an image into patches and treat them as a sequence, with no convolution at all. They beat convolutional networks at very large data scales and lose to them at small ones, because the convolution’s assumptions (locality and translation equivariance) are genuine prior knowledge that a transformer must learn from data instead. Hybrid designs now dominate.

That trade is a good general lesson. An architecture with built-in assumptions needs less data when the assumptions hold, and hits a ceiling the more flexible one does not.

What you will actually do

Almost nobody trains a vision network from scratch. You take a pretrained one and adapt it, which is the last lesson in this module. Free pretrained weights for hundreds of architectures are in timm and torchvision, and a fine-tuned ResNet on a few hundred images will beat anything you build from nothing on a free GPU.