Self-supervised learning: making labels out of the data itself

The trick that changed the field

Labels are expensive. Raw data is abundant. Self-supervised learning bridges the gap by constructing a supervised task out of unlabelled data, using part of the data as the label for the rest.

Hide a word and predict it from its context. Mask a patch of an image and reconstruct it. Take two crops of the same photograph and train the model to recognise that they came from the same photograph. In every case the supervision comes from the data’s own structure, so the training set is as large as the internet.

The representation learned this way transfers. A model that has learned to predict missing words has necessarily learned a great deal about language, and that knowledge is available for any downstream task with only a few hundred labels.

The three main shapes

Masked prediction. Hide part of the input, predict it. BERT masks 15% of tokens; masked autoencoders hide up to 75% of image patches. The task is trivially generated and forces the model to learn context.

Next-item prediction. Predict what comes next from what came before. This is how language models are trained, and it is the same objective as the word2vec idea scaled up enormously.

Contrastive learning. Create two augmented views of the same item (two crops, two colour distortions) and train so those two land close together while views of different items land apart. SimCLR and MoCo established this for images. The subtlety is that the augmentations define what the model is allowed to consider irrelevant: if you augment with colour jitter, the model learns that colour does not matter, which is wrong for a task about ripeness of fruit.

That last point is the general lesson. The pretext task decides what the representation encodes, and choosing it is where the design judgement lives.

Semi-supervised learning: a few labels plus a lot of data

A related and more immediately usable idea when you have 500 labels and 50,000 unlabelled rows.

Self-training / pseudo-labelling. Train on the labels you have, predict on the unlabelled data, add the most confident predictions as labels, retrain. It works when the initial model is decent and confidence is meaningful. Its failure mode is confirmation: a confident early mistake becomes a label and the model learns it thoroughly. Use a high confidence threshold, add data gradually, and check whether performance on your genuine validation set actually improves at each round.

Consistency regularisation. Require the model to give the same output for two perturbed versions of the same unlabelled input. This uses the unlabelled data without ever guessing a label for it, which avoids the confirmation problem, and it is the basis of most modern methods in this area.

Label propagation. Build a graph of similar rows and spread labels along it. Available in scikit-learn as LabelSpreading, and effective when your data does form clusters that align with the classes.

Active learning: choosing what to label

If labelling costs money, choose carefully what to label next.

• Uncertainty sampling: label the rows the current model is least sure about, since those are nearest the decision boundary and most informative.
• Diversity sampling: label rows unlike anything already labelled, to cover the space.
• Query by committee: train several models and label the rows they disagree about most.

A practical loop: label 200 rows, train, score the unlabelled pool, take the 100 most uncertain and the 50 most different, label those, repeat. Compared with random labelling, this routinely reaches the same accuracy for half the labels, and the saving is real money.

The caveat: uncertainty sampling biases your labelled set towards boundary cases, so it is no longer a random sample of the population. That is fine for training and it means you cannot use it as a test set. Keep a separate randomly labelled set for evaluation, always.

What this means for a small team

The practical route for anyone with a real problem and no labels:

1. Start from a pretrained model: a sentence embedding model, an image model from timm, all free on Hugging Face. Somebody else paid for the self-supervised pretraining.
2. Label a few hundred examples using active learning to choose which.
3. Fine-tune, or just train a linear model on the embeddings. The second option needs no GPU and is often within a point or two of the first.
4. Keep a randomly sampled test set so you can tell whether any of it worked.

That sequence turns “we have no labelled data” from a blocking condition into an afternoon, and it is available to anyone with a laptop and an internet connection.