CNNs & transfer learning

Convolutional neural networks (CNNs) power most computer-vision systems in production, from quality control on a Tiruppur garment line to identifying crop pests from a farmer's phone photo. This lesson explains why convolutions work and shows how transfer learning lets you build accurate image classifiers with only a few hundred labelled images.

Why not a plain MLP?

A 224x224 RGB image has 150,528 values. Connecting each to 1,000 hidden neurons needs 150 million weights in one layer, ignores the fact that nearby pixels are related, and has to relearn "what an edge looks like" at every position. Convolutions fix all three problems.

How a convolution works

A filter (kernel), typically 3x3, slides across the image. At each position it computes a weighted sum of the pixels under it, producing a feature map. The same small set of weights is reused everywhere (weight sharing), so a filter that detects a vertical edge finds it anywhere in the image.

Concept	Meaning
Kernel size	Size of the sliding window (3x3 is standard)
Channels	Number of filters; each learns a different pattern
Stride	Step size; stride 2 halves height and width
Padding	Zeros around the border to preserve size
Pooling	Down-samples (max or average) to add robustness and cut compute
Receptive field	Area of the original image a neuron "sees" — grows with depth

Output size for one dimension: (input - kernel + 2*padding) / stride + 1. For a 64-pixel input, kernel 3, padding 1, stride 1, the output stays 64.

A small CNN from scratch

import torch
import torch.nn as nn

class SmallCNN(nn.Module):
    def __init__(self, n_classes: int = 10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.MaxPool2d(2),                                   # 28 -> 14
            nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
            nn.MaxPool2d(2),                                   # 14 -> 7
        )
        self.classifier = nn.Sequential(
            nn.Flatten(), nn.Dropout(0.3), nn.Linear(64 * 7 * 7, n_classes)
        )

    def forward(self, x):
        return self.classifier(self.features(x))

print(SmallCNN()(torch.rand(4, 1, 28, 28)).shape)   # torch.Size([4, 10])

On Fashion-MNIST this reaches about 91–92% test accuracy — clearly better than the MLP, with fewer parameters.

Famous architectures

• ResNet introduced residual (skip) connections, letting networks reach 50–150 layers.
• EfficientNet and MobileNet balance accuracy and speed; MobileNet suits phones.
• ConvNeXt and Vision Transformers (ViT) are modern high-accuracy choices.

Transfer learning

Models trained on ImageNet (1.2 million images, 1,000 classes) have already learned general visual features: edges, textures, shapes. You can reuse them:

1. Feature extraction – freeze the backbone, train only a new final layer. Best for small datasets (a few hundred images).
2. Fine-tuning – after the head is trained, unfreeze some or all layers and train with a small learning rate. Best when you have thousands of images or your images differ from ImageNet (X-rays, satellite imagery).

from torchvision import models

weights = models.ResNet18_Weights.DEFAULT
model = models.resnet18(weights=weights)

for p in model.parameters():          # stage 1: freeze everything
    p.requires_grad = False

n_classes = 5                         # e.g. five sweet varieties
model.fc = nn.Linear(model.fc.in_features, n_classes)   # new head is trainable

optimizer = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
preprocess = weights.transforms()     # the exact resize/normalise the backbone expects

After a few epochs, unfreeze the last block and fine-tune gently:

for p in model.layer4.parameters():
    p.requires_grad = True

optimizer = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
], weight_decay=1e-2)

Using discriminative learning rates — smaller for early layers, larger for the head — protects the useful pretrained features.

Choosing a strategy

Your data	Similar to everyday photos?	Strategy
Under 1,000 images	Yes	Freeze backbone, train head
1,000–10,000 images	Yes	Train head, then unfreeze last blocks
Over 10,000 images	No (medical, satellite)	Fine-tune the whole network

Looking inside the model

Tools like Grad-CAM highlight which image regions drove a prediction. If a "jalebi" prediction focuses on the shop's orange tray rather than the sweet itself, your data has a shortcut you need to fix by collecting more varied photos.

Common mistakes

• Using pretrained weights without the matching normalisation.
• Fine-tuning the whole network with a high learning rate, wiping out pretrained features.
• Forgetting that frozen batch-norm layers still update running stats in train() mode; for tiny datasets keep the backbone in eval() mode.

Key takeaways

• Convolutions exploit locality and weight sharing, making image models efficient.
• Pretrained backbones plus a new head give strong results with little data.
• Freeze first, then unfreeze gradually with smaller learning rates.