Standing on somebody else’s compute

Why you should not train from scratch

Training a vision model from random weights needs hundreds of thousands of labelled images. Training a language model from scratch needs hundreds of billions of tokens and a compute budget in the millions.

You do not have either. You do not need either, because someone already did it and released the weights.

Transfer learning takes a model trained on a large general dataset and adapts it to your specific task with a few hundred or few thousand examples. It is the single most practically important technique in this module, and it is what makes deep learning available to someone with a laptop.

Why it works

The convolution lesson showed what the early layers learn: edges, textures, shapes. None of that is specific to the thousand ImageNet categories. Edges are edges whether the image is a dog, a chest X-ray, or a defective weld.

So the early layers are reusable and only the final task-specific layers need replacing. The same holds in language: a model that learned to predict the next word has learned grammar, common sense and world knowledge that no task-specific dataset could teach it.

Three levels of adaptation

1. Feature extraction. Freeze the whole pretrained model, use it to produce vectors, train a small classifier on those.

import timm, torch
backbone = timm.create_model("resnet50", pretrained=True, num_classes=0)
backbone.train(False)                 # evaluation mode: no dropout, fixed norm stats
with torch.no_grad():
    feats = backbone(images)          # (n, 2048)
# then: LogisticRegression().fit(feats, y)

Fastest, needs the least data, runs comfortably on a CPU, and cannot overfit the backbone because the backbone never changes. On a few hundred images this is often within a couple of points of full fine-tuning, and it is what to try first.

2. Fine-tune the last layers. Freeze the early layers, train the last block plus a new head with a low learning rate. A middle option when your data resembles the pretraining data but not exactly.

3. Fine-tune everything. Train all weights with a small learning rate, typically 10 to 100 times smaller than you would use from scratch. Best results with enough data; risks destroying the pretrained features if the learning rate is too high, which is called catastrophic forgetting and looks like the model getting suddenly and permanently worse in the first epoch.

Discriminative learning rates (a lower rate for early layers and a higher one for later ones) split the difference, on the reasoning that early features need less change than late ones. It is standard practice in fastai and worth knowing.

Parameter-efficient fine-tuning

For large language models, fine-tuning every weight requires holding the model, its gradients and the optimiser state in memory at once: far more than a free GPU has.

LoRA freezes the original weights and trains small low-rank matrices added alongside them. It typically trains under 1% of the parameters, reaches close to full fine-tuning quality, and the resulting adapter is a few megabytes rather than gigabytes. QLoRA adds 4-bit quantisation of the frozen base model, which brings fine-tuning a 7-billion-parameter model within reach of a single free-tier GPU.

The free library is peft from Hugging Face, and this is the route by which a person with no budget can adapt a large model.

Where to get models

Hugging Face Hub: hundreds of thousands of models, free. timm: vision models. torchvision and Keras Applications: the classics, bundled.

Check the licence before you deploy. It varies from fully permissive to research-only to “open weights with restrictions on use above a certain scale”, and the terms are not always what the word “open” suggests. This matters commercially and it is a genuine area of dispute: several licences described as open are not open by the standard definitions, and the disagreement is unresolved.

When transfer learning does not help

• Your domain is very unlike the pretraining data. ImageNet features transfer poorly to spectrograms, satellite imagery or scientific plots. It usually still beats random initialisation, and by less than you expect.
• Tabular data. There is no equivalent of ImageNet for tables, because there is no shared structure across datasets to transfer. Foundation models for tabular data are an active research area (TabPFN is the most notable) and gradient boosting remains the practical answer.
• You have millions of labels already. Then training from scratch is viable and may fit your domain better.

The honest recipe for someone with a laptop

1. Find a pretrained model close to your domain on Hugging Face.
2. Extract features and fit logistic regression. Record the number.
3. Fine-tune the last block. Record the number.
4. Fine-tune everything with a low rate, if step 3 helped and you have the data.
5. Stop at whichever step stopped improving.

Most people skip to step 4 and are surprised when it does worse than step 2 on 300 images. The steps are in this order because each one needs more data than the last, and the measurement tells you which one your data supports.