{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"transfer-learning","title":"Standing on somebody else's compute","url":"https://addaly.com/learn/machine-learning-foundations/transfer-learning","position":82,"of":92,"minutes":9,"answer":"Pretrained early layers encode general structure that transfers across tasks, so start by freezing the backbone and fitting a linear model on its features.","keyIdea":"Pretrained early layers encode general structure that transfers across tasks, so start by freezing the backbone and fitting a linear model on its features, and escalate to full fine-tuning only when the measurement says your data supports it."},"body":{"format":"markdown","text":"## Why you should not train from scratch\n\nTraining a vision model from random weights needs hundreds of thousands of labelled images. Training a language model from scratch needs hundreds of billions of tokens and a compute budget in the millions.\n\nYou do not have either. You do not need either, because someone already did it and released the weights.\n\n**Transfer learning** takes a model trained on a large general dataset and adapts it to your specific task with a few hundred or few thousand examples. It is the single most practically important technique in this module, and it is what makes deep learning available to someone with a laptop.\n\n## Why it works\n\nThe convolution lesson showed what the early layers learn: edges, textures, shapes. None of that is specific to the thousand ImageNet categories. Edges are edges whether the image is a dog, a chest X-ray, or a defective weld.\n\nSo the early layers are reusable and only the final task-specific layers need replacing. The same holds in language: a model that learned to predict the next word has learned grammar, common sense and world knowledge that no task-specific dataset could teach it.\n\n## Three levels of adaptation\n\n**1. Feature extraction.** Freeze the whole pretrained model, use it to produce vectors, train a small classifier on those.\n\n```python\nimport timm, torch\nbackbone = timm.create_model(\"resnet50\", pretrained=True, num_classes=0)\nbackbone.train(False)                 # evaluation mode: no dropout, fixed norm stats\nwith torch.no_grad():\n    feats = backbone(images)          # (n, 2048)\n# then: LogisticRegression().fit(feats, y)\n```\n\nFastest, needs the least data, runs comfortably on a CPU, and cannot overfit the backbone because the backbone never changes. On a few hundred images this is often within a couple of points of full fine-tuning, and it is what to try first.\n\n**2. Fine-tune the last layers.** Freeze the early layers, train the last block plus a new head with a low learning rate. A middle option when your data resembles the pretraining data but not exactly.\n\n**3. Fine-tune everything.** Train all weights with a small learning rate, typically 10 to 100 times smaller than you would use from scratch. Best results with enough data; risks destroying the pretrained features if the learning rate is too high, which is called catastrophic forgetting and looks like the model getting suddenly and permanently worse in the first epoch.\n\n**Discriminative learning rates** (a lower rate for early layers and a higher one for later ones) split the difference, on the reasoning that early features need less change than late ones. It is standard practice in fastai and worth knowing.\n\n## Parameter-efficient fine-tuning\n\nFor large language models, fine-tuning every weight requires holding the model, its gradients and the optimiser state in memory at once: far more than a free GPU has.\n\n**LoRA** freezes the original weights and trains small low-rank matrices added alongside them. It typically trains under 1% of the parameters, reaches close to full fine-tuning quality, and the resulting adapter is a few megabytes rather than gigabytes. **QLoRA** adds 4-bit quantisation of the frozen base model, which brings fine-tuning a 7-billion-parameter model within reach of a single free-tier GPU.\n\nThe free library is `peft` from Hugging Face, and this is the route by which a person with no budget can adapt a large model.\n\n## Where to get models\n\n**Hugging Face Hub**: hundreds of thousands of models, free. **timm**: vision models. **torchvision** and **Keras Applications**: the classics, bundled.\n\nCheck the licence before you deploy. It varies from fully permissive to research-only to \"open weights with restrictions on use above a certain scale\", and the terms are not always what the word \"open\" suggests. This matters commercially and it is a genuine area of dispute: several licences described as open are not open by the standard definitions, and the disagreement is unresolved.\n\n## When transfer learning does not help\n\n- **Your domain is very unlike the pretraining data.** ImageNet features transfer poorly to spectrograms, satellite imagery or scientific plots. It usually still beats random initialisation, and by less than you expect.\n- **Tabular data.** There is no equivalent of ImageNet for tables, because there is no shared structure across datasets to transfer. Foundation models for tabular data are an active research area (TabPFN is the most notable) and gradient boosting remains the practical answer.\n- **You have millions of labels already.** Then training from scratch is viable and may fit your domain better.\n\n## The honest recipe for someone with a laptop\n\n1. Find a pretrained model close to your domain on Hugging Face.\n2. Extract features and fit logistic regression. Record the number.\n3. Fine-tune the last block. Record the number.\n4. Fine-tune everything with a low rate, if step 3 helped and you have the data.\n5. Stop at whichever step stopped improving.\n\nMost people skip to step 4 and are surprised when it does worse than step 2 on 300 images. The steps are in this order because each one needs more data than the last, and the measurement tells you which one your data supports.\n\n> Figure: Four ways to adapt a pretrained ResNet-50 to 300 labelled images. Bars (% accuracy on held-out images): Feature extraction, then logistic regression 91; Fine-tune the last block only 92.4; Fine-tune everything, rate 100× lower 90.6; Fine-tune everything, from-scratch rate 71.5. On this little data the frozen backbone with a logistic regression on its features is within two points of the best, and full fine-tuning at a from-scratch learning rate destroys the pretrained features in the first epoch. The steps are ordered by how much data each needs, and the measurement says which one yours supports."},"check":{"question":"A team fine-tunes all layers of a pretrained ResNet on 300 medical images at the same learning rate used for training from scratch. Accuracy is worse than a logistic regression on frozen features. Why?","options":[{"text":"Fine-tuning requires batch normalisation to be frozen, and leaving it active corrupted the running statistics.","correct":false,"tempting":"Batch norm statistics on small batches are a real fine-tuning hazard worth handling, but they are secondary to a learning rate orders of magnitude too high."},{"text":"ResNet features do not transfer to medical images, so the pretrained weights were useless from the start.","correct":false,"tempting":"Domain distance does reduce transfer, but the frozen-feature model outperformed the fine-tuned one using the same weights, so the features clearly carried value."},{"text":"300 images is below the minimum for any deep learning approach, so both results are unreliable.","correct":false,"tempting":"The frozen-feature approach works well at this scale, which is exactly why it is the recommended first step."},{"text":"A learning rate suited to random initialisation is far too large for pretrained weights, so the first epochs overwrite the general features the model had learned before the small dataset can teach it anything better.","correct":true}]},"prev":{"slug":"from-sequences-to-attention","title":"Sequences: from recurrence to attention","url":"https://addaly.com/learn/machine-learning-foundations/from-sequences-to-attention"},"next":{"slug":"the-model-is-not-the-system","title":"The model is about 5% of what you have to build","url":"https://addaly.com/learn/machine-learning-foundations/the-model-is-not-the-system"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Standing on somebody else's compute', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/transfer-learning (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Standing on somebody else’s compute","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"Pretrained early layers encode general structure that transfers across tasks, so start by freezing the backbone and fitting a linear model on its features, and escalate to full fine-tuning only when the measurement says your data supports it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% accuracy on held-out images","title":"Four ways to adapt a pretrained ResNet-50 to 300 labelled images","series":[{"label":"Feature extraction, then logistic regression","value":91,"highlight":true},{"label":"Fine-tune the last block only","value":92.4},{"label":"Fine-tune everything, rate 100× lower","value":90.6},{"label":"Fine-tune everything, from-scratch rate","value":71.5}],"caption":"On this little data the frozen backbone with a logistic regression on its features is within two points of the best, and full fine-tuning at a from-scratch learning rate destroys the pretrained features in the first epoch. The steps are ordered by how much data each needs, and the measurement says which one yours supports."},"alt":"Figure: Four ways to adapt a pretrained ResNet-50 to 300 labelled images. Bars (% accuracy on held-out images): Feature extraction, then logistic regression 91; Fine-tune the last block only 92.4; Fine-tune everything, rate 100× lower 90.6; Fine-tune everything, from-scratch rate 71.5. On this little data the frozen backbone with a logistic regression on its features is within two points of the best, and full fine-tuning at a from-scratch learning rate destroys the pretrained features in the first epoch. The steps are ordered by how much data each needs, and the measurement says which one yours supports."}},"blocks":[{"t":"h2","id":"why-you-should-not-train-from-scratch","runs":[{"text":"Why you should not train from scratch"}]},{"t":"p","runs":[{"text":"Training a vision model from random weights needs hundreds of thousands of labelled images. Training a language model from scratch needs hundreds of billions of tokens and a compute budget in the millions."}]},{"t":"p","runs":[{"text":"You do not have either. You do not need either, because someone already did it and released the weights."}]},{"t":"p","runs":[{"text":"Transfer learning","b":true},{"text":" takes a model trained on a large general dataset and adapts it to your specific task with a few hundred or few thousand examples. It is the single most practically important technique in this module, and it is what makes deep learning available to someone with a laptop."}]},{"t":"h2","id":"why-it-works","runs":[{"text":"Why it works"}]},{"t":"p","runs":[{"text":"The convolution lesson showed what the early layers learn: edges, textures, shapes. None of that is specific to the thousand ImageNet categories. Edges are edges whether the image is a dog, a chest X-ray, or a defective weld."}]},{"t":"p","runs":[{"text":"So the early layers are reusable and only the final task-specific layers need replacing. The same holds in language: a model that learned to predict the next word has learned grammar, common sense and world knowledge that no task-specific dataset could teach it."}]},{"t":"h2","id":"three-levels-of-adaptation","runs":[{"text":"Three levels of adaptation"}]},{"t":"p","runs":[{"text":"1. Feature extraction.","b":true},{"text":" Freeze the whole pretrained model, use it to produce vectors, train a small classifier on those."}]},{"t":"code","lang":"python","label":"Python","text":"import timm, torch\nbackbone = timm.create_model(\"resnet50\", pretrained=True, num_classes=0)\nbackbone.train(False)                 # evaluation mode: no dropout, fixed norm stats\nwith torch.no_grad():\n    feats = backbone(images)          # (n, 2048)\n# then: LogisticRegression().fit(feats, y)"},{"t":"p","runs":[{"text":"Fastest, needs the least data, runs comfortably on a CPU, and cannot overfit the backbone because the backbone never changes. On a few hundred images this is often within a couple of points of full fine-tuning, and it is what to try first."}]},{"t":"p","runs":[{"text":"2. Fine-tune the last layers.","b":true},{"text":" Freeze the early layers, train the last block plus a new head with a low learning rate. A middle option when your data resembles the pretraining data but not exactly."}]},{"t":"p","runs":[{"text":"3. Fine-tune everything.","b":true},{"text":" Train all weights with a small learning rate, typically 10 to 100 times smaller than you would use from scratch. Best results with enough data; risks destroying the pretrained features if the learning rate is too high, which is called catastrophic forgetting and looks like the model getting suddenly and permanently worse in the first epoch."}]},{"t":"p","runs":[{"text":"Discriminative learning rates","b":true},{"text":" (a lower rate for early layers and a higher one for later ones) split the difference, on the reasoning that early features need less change than late ones. It is standard practice in fastai and worth knowing."}]},{"t":"h2","id":"parameter-efficient-fine-tuning","runs":[{"text":"Parameter-efficient fine-tuning"}]},{"t":"p","runs":[{"text":"For large language models, fine-tuning every weight requires holding the model, its gradients and the optimiser state in memory at once: far more than a free GPU has."}]},{"t":"p","runs":[{"text":"LoRA","b":true},{"text":" freezes the original weights and trains small low-rank matrices added alongside them. It typically trains under 1% of the parameters, reaches close to full fine-tuning quality, and the resulting adapter is a few megabytes rather than gigabytes. "},{"text":"QLoRA","b":true},{"text":" adds 4-bit quantisation of the frozen base model, which brings fine-tuning a 7-billion-parameter model within reach of a single free-tier GPU."}]},{"t":"p","runs":[{"text":"The free library is "},{"text":"peft","code":true},{"text":" from Hugging Face, and this is the route by which a person with no budget can adapt a large model."}]},{"t":"h2","id":"where-to-get-models","runs":[{"text":"Where to get models"}]},{"t":"p","runs":[{"text":"Hugging Face Hub","b":true},{"text":": hundreds of thousands of models, free. "},{"text":"timm","b":true},{"text":": vision models. "},{"text":"torchvision","b":true},{"text":" and "},{"text":"Keras Applications","b":true},{"text":": the classics, bundled."}]},{"t":"p","runs":[{"text":"Check the licence before you deploy. It varies from fully permissive to research-only to “open weights with restrictions on use above a certain scale”, and the terms are not always what the word “open” suggests. This matters commercially and it is a genuine area of dispute: several licences described as open are not open by the standard definitions, and the disagreement is unresolved."}]},{"t":"h2","id":"when-transfer-learning-does-not-help","runs":[{"text":"When transfer learning does not help"}]},{"t":"ul","items":[[{"text":"Your domain is very unlike the pretraining data.","b":true},{"text":" ImageNet features transfer poorly to spectrograms, satellite imagery or scientific plots. It usually still beats random initialisation, and by less than you expect."}],[{"text":"Tabular data.","b":true},{"text":" There is no equivalent of ImageNet for tables, because there is no shared structure across datasets to transfer. Foundation models for tabular data are an active research area (TabPFN is the most notable) and gradient boosting remains the practical answer."}],[{"text":"You have millions of labels already.","b":true},{"text":" Then training from scratch is viable and may fit your domain better."}]]},{"t":"h2","id":"the-honest-recipe-for-someone-with-a-laptop","runs":[{"text":"The honest recipe for someone with a laptop"}]},{"t":"ol","start":1,"items":[[{"text":"Find a pretrained model close to your domain on Hugging Face."}],[{"text":"Extract features and fit logistic regression. Record the number."}],[{"text":"Fine-tune the last block. Record the number."}],[{"text":"Fine-tune everything with a low rate, if step 3 helped and you have the data."}],[{"text":"Stop at whichever step stopped improving."}]]},{"t":"p","runs":[{"text":"Most people skip to step 4 and are surprised when it does worse than step 2 on 300 images. The steps are in this order because each one needs more data than the last, and the measurement tells you which one your data supports."}]}],"check":{"question":[{"text":"A team fine-tunes all layers of a pretrained ResNet on 300 medical images at the same learning rate used for training from scratch. Accuracy is worse than a logistic regression on frozen features. Why?"}],"options":[{"text":[{"text":"Fine-tuning requires batch normalisation to be frozen, and leaving it active corrupted the running statistics."}],"correct":false,"tempting":[{"text":"Batch norm statistics on small batches are a real fine-tuning hazard worth handling, but they are secondary to a learning rate orders of magnitude too high."}]},{"text":[{"text":"ResNet features do not transfer to medical images, so the pretrained weights were useless from the start."}],"correct":false,"tempting":[{"text":"Domain distance does reduce transfer, but the frozen-feature model outperformed the fine-tuned one using the same weights, so the features clearly carried value."}]},{"text":[{"text":"300 images is below the minimum for any deep learning approach, so both results are unreliable."}],"correct":false,"tempting":[{"text":"The frozen-feature approach works well at this scale, which is exactly why it is the recommended first step."}]},{"text":[{"text":"A learning rate suited to random initialisation is far too large for pretrained weights, so the first epochs overwrite the general features the model had learned before the small dataset can teach it anything better."}],"correct":true,"tempting":null}],"order":[2,0,1,3],"why":null},"prev":{"slug":"from-sequences-to-attention","title":"Sequences: from recurrence to attention","minutes":9},"next":{"slug":"the-model-is-not-the-system","title":"The model is about 5% of what you have to build","minutes":8}}}