Fine-tuning a classifier properly

The setup

Module one argued that a fixed set of labels wants an encoder rather than a generative model. This is how that is done.

from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL = "answerdotai/ModernBERT-base"     # 149M; or microsoft/deberta-v3-base
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForSequenceClassification.from_pretrained(
    MODEL, num_labels=6, id2label=ID2LABEL, label2id=LABEL2ID)

Pass id2label and label2id. They are saved in the config, which means the model that comes back off disk in six months says what its outputs mean. Without them you have a tensor of six numbers and a note in somebody’s head.

Full fine-tuning at this size costs about 6 GB of GPU memory at a learning rate of 2e-5 for 3 to 5 epochs. No adapters, no quantisation; the model is small enough that all the machinery from earlier modules is unnecessary.

Class imbalance, which you will have

Real label distributions are not uniform. A six-way routing task might be 62% one class and 3% another. Trained naively, the model learns to predict the majority class and scores 62% accuracy while being useless.

Three treatments, in order of preference:

Weighted loss. Weight each class inversely to its frequency. One override:

weights = torch.tensor(len(y) / (n_classes * np.bincount(y)), dtype=torch.float)
loss = F.cross_entropy(logits, labels, weight=weights.to(logits.device))

Threshold adjustment after training. Train normally, then move the decision boundary per class using a validation set. Often better than reweighting, because it separates “what does the model believe” from “what do we do about it”, and lets you change the second without retraining.

Resampling. Oversample the minority or undersample the majority. Simple, and oversampling duplicates examples, which invites memorisation of your rarest and most precious class.

Measure with macro F1, not accuracy

Accuracy is dominated by the majority class and will tell you a useless model is a good one. Macro-averaged F1 weights every class equally, so the 3% class counts as much as the 62% one.

Print the per-class report, always:

from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(y_true, y_pred, target_names=LABELS, digits=3))
print(confusion_matrix(y_true, y_pred))

Read the confusion matrix rather than the summary line. It tells you which classes are being confused with which, and that is almost always actionable: two labels that the model mixes up are usually two labels your annotators mixed up, which is a taxonomy problem rather than a model problem.

Multi-label is a different loss

If an item can carry several labels at once, the labels are not mutually exclusive and softmax is wrong. Use independent sigmoids:

model = AutoModelForSequenceClassification.from_pretrained(
    MODEL, num_labels=12, problem_type="multi_label_classification")

That problem_type switches the loss to binary cross-entropy per label. Then tune a threshold per label on validation data: a single global 0.5 is rarely right when label frequencies differ by an order of magnitude.

Calibration, if you will route on confidence

A common and sensible design is to use the classifier for confident cases and fall back to an expensive model for the rest. That rule depends on the confidence meaning something.

Fine-tuned classifiers are typically overconfident: predictions cluster near 0 and 1, and those at 0.9 are right less often than 90% of the time. Check it by binning held-out predictions by confidence and comparing each bin’s mean confidence with its accuracy. Temperature scaling (one scalar fitted on validation data, dividing the logits) corrects most of the gap in about ten lines.

Knowing when you have enough data

Plot the learning curve. Train on 25%, 50%, 75% and 100% of your labels and plot macro F1 against training size.

• Still climbing steeply at 100%: more labels is your best investment.
• Flattening: more labels will not help much; better labels, better features or a different taxonomy might.

This takes four short runs on a 150M model (under an hour on free hardware) and it answers the question teams usually argue about instead: whether to spend the next week labelling.

Where the encoder stops being right

Be honest about the boundary. An encoder classifier needs labels, cannot take a new class without retraining, and degrades silently when your inputs drift. If your label set changes monthly, or you have fewer than a few hundred examples per class, prompting a general model is the better tool despite costing more per call.