{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"learning-from-unlabelled-data","title":"Self-supervised learning: making labels out of the data itself","url":"https://addaly.com/learn/machine-learning-foundations/learning-from-unlabelled-data","position":72,"of":92,"minutes":9,"answer":"Self-supervised learning manufactures a supervised task from the data's own structure, and the pretext task chosen decides what the representation encodes.","keyIdea":"Self-supervised learning manufactures a supervised task from the data's own structure, and the pretext task chosen decides what the representation encodes, which is why a model pretrained with colour augmentation has learned that colour does not matter."},"body":{"format":"markdown","text":"## The trick that changed the field\n\nLabels are expensive. Raw data is abundant. Self-supervised learning bridges the gap by constructing a supervised task out of unlabelled data, using part of the data as the label for the rest.\n\nHide a word and predict it from its context. Mask a patch of an image and reconstruct it. Take two crops of the same photograph and train the model to recognise that they came from the same photograph. In every case the supervision comes from the data's own structure, so the training set is as large as the internet.\n\nThe representation learned this way transfers. A model that has learned to predict missing words has necessarily learned a great deal about language, and that knowledge is available for any downstream task with only a few hundred labels.\n\n## The three main shapes\n\n**Masked prediction.** Hide part of the input, predict it. BERT masks 15% of tokens; masked autoencoders hide up to 75% of image patches. The task is trivially generated and forces the model to learn context.\n\n**Next-item prediction.** Predict what comes next from what came before. This is how language models are trained, and it is the same objective as the word2vec idea scaled up enormously.\n\n**Contrastive learning.** Create two augmented views of the same item (two crops, two colour distortions) and train so those two land close together while views of different items land apart. SimCLR and MoCo established this for images. The subtlety is that the augmentations define what the model is allowed to consider irrelevant: if you augment with colour jitter, the model learns that colour does not matter, which is wrong for a task about ripeness of fruit.\n\nThat last point is the general lesson. **The pretext task decides what the representation encodes**, and choosing it is where the design judgement lives.\n\n## Semi-supervised learning: a few labels plus a lot of data\n\nA related and more immediately usable idea when you have 500 labels and 50,000 unlabelled rows.\n\n**Self-training / pseudo-labelling.** Train on the labels you have, predict on the unlabelled data, add the most confident predictions as labels, retrain. It works when the initial model is decent and confidence is meaningful. Its failure mode is confirmation: a confident early mistake becomes a label and the model learns it thoroughly. Use a high confidence threshold, add data gradually, and check whether performance on your genuine validation set actually improves at each round.\n\n**Consistency regularisation.** Require the model to give the same output for two perturbed versions of the same unlabelled input. This uses the unlabelled data without ever guessing a label for it, which avoids the confirmation problem, and it is the basis of most modern methods in this area.\n\n**Label propagation.** Build a graph of similar rows and spread labels along it. Available in scikit-learn as `LabelSpreading`, and effective when your data does form clusters that align with the classes.\n\n## Active learning: choosing what to label\n\nIf labelling costs money, choose carefully what to label next.\n\n- **Uncertainty sampling**: label the rows the current model is least sure about, since those are nearest the decision boundary and most informative.\n- **Diversity sampling**: label rows unlike anything already labelled, to cover the space.\n- **Query by committee**: train several models and label the rows they disagree about most.\n\nA practical loop: label 200 rows, train, score the unlabelled pool, take the 100 most uncertain and the 50 most different, label those, repeat. Compared with random labelling, this routinely reaches the same accuracy for half the labels, and the saving is real money.\n\nThe caveat: uncertainty sampling biases your labelled set towards boundary cases, so it is no longer a random sample of the population. That is fine for training and it means you cannot use it as a test set. Keep a separate randomly labelled set for evaluation, always.\n\n## What this means for a small team\n\nThe practical route for anyone with a real problem and no labels:\n\n1. **Start from a pretrained model**: a sentence embedding model, an image model from `timm`, all free on Hugging Face. Somebody else paid for the self-supervised pretraining.\n2. **Label a few hundred examples** using active learning to choose which.\n3. **Fine-tune, or just train a linear model on the embeddings.** The second option needs no GPU and is often within a point or two of the first.\n4. **Keep a randomly sampled test set** so you can tell whether any of it worked.\n\nThat sequence turns \"we have no labelled data\" from a blocking condition into an afternoon, and it is available to anyone with a laptop and an internet connection."},"check":{"question":"A team uses uncertainty-based active learning to select 800 rows for labelling, then splits those 800 into train and test. Their test accuracy is far below production performance. Why?","options":[{"text":"800 rows is too few for a reliable test split, so the estimate is dominated by variance.","correct":false,"tempting":"A small test set is imprecise, but imprecision produces scatter in both directions rather than a consistent underestimate."},{"text":"Active learning selected rows the model was least certain about, so the labelled pool is concentrated near the decision boundary and is systematically harder than the real population.","correct":true},{"text":"Uncertainty sampling introduces label noise, since uncertain rows are harder for annotators to label correctly.","correct":false,"tempting":"Boundary cases can indeed be harder to annotate, but label noise would degrade training and testing alike rather than creating this specific gap."},{"text":"The model was trained on rows it had already scored, which constitutes leakage into the test set.","correct":false,"tempting":"Scoring unlabelled rows to select them does not expose their labels, so no target information leaks through the selection step."}]},"prev":{"slug":"finding-what-does-not-belong","title":"Anomaly detection, and the base rate that ruins it","url":"https://addaly.com/learn/machine-learning-foundations/finding-what-does-not-belong"},"next":{"slug":"neural-networks","title":"Neural networks as the natural next step","url":"https://addaly.com/learn/machine-learning-foundations/neural-networks"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Self-supervised learning: making labels out of the data itself', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/learning-from-unlabelled-data (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Self-supervised learning: making labels out of the data itself","courseTitle":"Machine Learning, Foundations","module":{"index":7,"slug":"learning-without-labels","title":"Learning without labels, and learning a representation"},"answer":null,"keyIdea":"Self-supervised learning manufactures a supervised task from the data’s own structure, and the pretext task chosen decides what the representation encodes, which is why a model pretrained with colour augmentation has learned that colour does not matter.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-trick-that-changed-the-field","runs":[{"text":"The trick that changed the field"}]},{"t":"p","runs":[{"text":"Labels are expensive. Raw data is abundant. Self-supervised learning bridges the gap by constructing a supervised task out of unlabelled data, using part of the data as the label for the rest."}]},{"t":"p","runs":[{"text":"Hide a word and predict it from its context. Mask a patch of an image and reconstruct it. Take two crops of the same photograph and train the model to recognise that they came from the same photograph. In every case the supervision comes from the data’s own structure, so the training set is as large as the internet."}]},{"t":"p","runs":[{"text":"The representation learned this way transfers. A model that has learned to predict missing words has necessarily learned a great deal about language, and that knowledge is available for any downstream task with only a few hundred labels."}]},{"t":"h2","id":"the-three-main-shapes","runs":[{"text":"The three main shapes"}]},{"t":"p","runs":[{"text":"Masked prediction.","b":true},{"text":" Hide part of the input, predict it. BERT masks 15% of tokens; masked autoencoders hide up to 75% of image patches. The task is trivially generated and forces the model to learn context."}]},{"t":"p","runs":[{"text":"Next-item prediction.","b":true},{"text":" Predict what comes next from what came before. This is how language models are trained, and it is the same objective as the word2vec idea scaled up enormously."}]},{"t":"p","runs":[{"text":"Contrastive learning.","b":true},{"text":" Create two augmented views of the same item (two crops, two colour distortions) and train so those two land close together while views of different items land apart. SimCLR and MoCo established this for images. The subtlety is that the augmentations define what the model is allowed to consider irrelevant: if you augment with colour jitter, the model learns that colour does not matter, which is wrong for a task about ripeness of fruit."}]},{"t":"p","runs":[{"text":"That last point is the general lesson. "},{"text":"The pretext task decides what the representation encodes","b":true},{"text":", and choosing it is where the design judgement lives."}]},{"t":"h2","id":"semi-supervised-learning-a-few-labels-plus-a-lot-of-data","runs":[{"text":"Semi-supervised learning: a few labels plus a lot of data"}]},{"t":"p","runs":[{"text":"A related and more immediately usable idea when you have 500 labels and 50,000 unlabelled rows."}]},{"t":"p","runs":[{"text":"Self-training / pseudo-labelling.","b":true},{"text":" Train on the labels you have, predict on the unlabelled data, add the most confident predictions as labels, retrain. It works when the initial model is decent and confidence is meaningful. Its failure mode is confirmation: a confident early mistake becomes a label and the model learns it thoroughly. Use a high confidence threshold, add data gradually, and check whether performance on your genuine validation set actually improves at each round."}]},{"t":"p","runs":[{"text":"Consistency regularisation.","b":true},{"text":" Require the model to give the same output for two perturbed versions of the same unlabelled input. This uses the unlabelled data without ever guessing a label for it, which avoids the confirmation problem, and it is the basis of most modern methods in this area."}]},{"t":"p","runs":[{"text":"Label propagation.","b":true},{"text":" Build a graph of similar rows and spread labels along it. Available in scikit-learn as "},{"text":"LabelSpreading","code":true},{"text":", and effective when your data does form clusters that align with the classes."}]},{"t":"h2","id":"active-learning-choosing-what-to-label","runs":[{"text":"Active learning: choosing what to label"}]},{"t":"p","runs":[{"text":"If labelling costs money, choose carefully what to label next."}]},{"t":"ul","items":[[{"text":"Uncertainty sampling","b":true},{"text":": label the rows the current model is least sure about, since those are nearest the decision boundary and most informative."}],[{"text":"Diversity sampling","b":true},{"text":": label rows unlike anything already labelled, to cover the space."}],[{"text":"Query by committee","b":true},{"text":": train several models and label the rows they disagree about most."}]]},{"t":"p","runs":[{"text":"A practical loop: label 200 rows, train, score the unlabelled pool, take the 100 most uncertain and the 50 most different, label those, repeat. Compared with random labelling, this routinely reaches the same accuracy for half the labels, and the saving is real money."}]},{"t":"p","runs":[{"text":"The caveat: uncertainty sampling biases your labelled set towards boundary cases, so it is no longer a random sample of the population. That is fine for training and it means you cannot use it as a test set. Keep a separate randomly labelled set for evaluation, always."}]},{"t":"h2","id":"what-this-means-for-a-small-team","runs":[{"text":"What this means for a small team"}]},{"t":"p","runs":[{"text":"The practical route for anyone with a real problem and no labels:"}]},{"t":"ol","start":1,"items":[[{"text":"Start from a pretrained model","b":true},{"text":": a sentence embedding model, an image model from "},{"text":"timm","code":true},{"text":", all free on Hugging Face. Somebody else paid for the self-supervised pretraining."}],[{"text":"Label a few hundred examples","b":true},{"text":" using active learning to choose which."}],[{"text":"Fine-tune, or just train a linear model on the embeddings.","b":true},{"text":" The second option needs no GPU and is often within a point or two of the first."}],[{"text":"Keep a randomly sampled test set","b":true},{"text":" so you can tell whether any of it worked."}]]},{"t":"p","runs":[{"text":"That sequence turns “we have no labelled data” from a blocking condition into an afternoon, and it is available to anyone with a laptop and an internet connection."}]}],"check":{"question":[{"text":"A team uses uncertainty-based active learning to select 800 rows for labelling, then splits those 800 into train and test. Their test accuracy is far below production performance. Why?"}],"options":[{"text":[{"text":"800 rows is too few for a reliable test split, so the estimate is dominated by variance."}],"correct":false,"tempting":[{"text":"A small test set is imprecise, but imprecision produces scatter in both directions rather than a consistent underestimate."}]},{"text":[{"text":"Active learning selected rows the model was least certain about, so the labelled pool is concentrated near the decision boundary and is systematically harder than the real population."}],"correct":true,"tempting":null},{"text":[{"text":"Uncertainty sampling introduces label noise, since uncertain rows are harder for annotators to label correctly."}],"correct":false,"tempting":[{"text":"Boundary cases can indeed be harder to annotate, but label noise would degrade training and testing alike rather than creating this specific gap."}]},{"text":[{"text":"The model was trained on rows it had already scored, which constitutes leakage into the test set."}],"correct":false,"tempting":[{"text":"Scoring unlabelled rows to select them does not expose their labels, so no target information leaks through the selection step."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"finding-what-does-not-belong","title":"Anomaly detection, and the base rate that ruins it","minutes":8},"next":{"slug":"neural-networks","title":"Neural networks as the natural next step","minutes":9}}}