{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"fine-tuning-a-classifier","title":"Fine-tuning a classifier properly","url":"https://addaly.com/learn/fine-tuning/fine-tuning-a-classifier","position":60,"of":84,"minutes":9,"answer":"A 150M encoder with id2label in its config, a weighted loss or adjusted thresholds for imbalance, macro F1 with the confusion matrix read rather than the summary line.","keyIdea":"A 150M encoder with id2label in its config, a weighted loss or adjusted thresholds for imbalance, macro F1 with the confusion matrix read rather than the summary line, and a four-point learning curve that tells you whether another week of labelling is worth it."},"body":{"format":"markdown","text":"## The setup\n\nModule one argued that a fixed set of labels wants an encoder rather than a generative model. This is how that is done.\n\n```python\nfrom transformers import AutoModelForSequenceClassification, AutoTokenizer\n\nMODEL = \"answerdotai/ModernBERT-base\"     # 149M; or microsoft/deberta-v3-base\ntok = AutoTokenizer.from_pretrained(MODEL)\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    MODEL, num_labels=6, id2label=ID2LABEL, label2id=LABEL2ID)\n```\n\nPass `id2label` and `label2id`. They are saved in the config, which means the model that comes back off disk in six months says what its outputs mean. Without them you have a tensor of six numbers and a note in somebody's head.\n\nFull fine-tuning at this size costs about 6 GB of GPU memory at a learning rate of 2e-5 for 3 to 5 epochs. No adapters, no quantisation; the model is small enough that all the machinery from earlier modules is unnecessary.\n\n## Class imbalance, which you will have\n\nReal label distributions are not uniform. A six-way routing task might be 62% one class and 3% another. Trained naively, the model learns to predict the majority class and scores 62% accuracy while being useless.\n\nThree treatments, in order of preference:\n\n**Weighted loss.** Weight each class inversely to its frequency. One override:\n\n```python\nweights = torch.tensor(len(y) / (n_classes * np.bincount(y)), dtype=torch.float)\nloss = F.cross_entropy(logits, labels, weight=weights.to(logits.device))\n```\n\n**Threshold adjustment after training.** Train normally, then move the decision boundary per class using a validation set. Often better than reweighting, because it separates \"what does the model believe\" from \"what do we do about it\", and lets you change the second without retraining.\n\n**Resampling.** Oversample the minority or undersample the majority. Simple, and oversampling duplicates examples, which invites memorisation of your rarest and most precious class.\n\n## Measure with macro F1, not accuracy\n\nAccuracy is dominated by the majority class and will tell you a useless model is a good one. Macro-averaged F1 weights every class equally, so the 3% class counts as much as the 62% one.\n\nPrint the per-class report, always:\n\n```python\nfrom sklearn.metrics import classification_report, confusion_matrix\nprint(classification_report(y_true, y_pred, target_names=LABELS, digits=3))\nprint(confusion_matrix(y_true, y_pred))\n```\n\nRead the confusion matrix rather than the summary line. It tells you *which* classes are being confused with which, and that is almost always actionable: two labels that the model mixes up are usually two labels your annotators mixed up, which is a taxonomy problem rather than a model problem.\n\n## Multi-label is a different loss\n\nIf an item can carry several labels at once, the labels are not mutually exclusive and softmax is wrong. Use independent sigmoids:\n\n```python\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    MODEL, num_labels=12, problem_type=\"multi_label_classification\")\n```\n\nThat `problem_type` switches the loss to binary cross-entropy per label. Then tune a threshold **per label** on validation data: a single global 0.5 is rarely right when label frequencies differ by an order of magnitude.\n\n## Calibration, if you will route on confidence\n\nA common and sensible design is to use the classifier for confident cases and fall back to an expensive model for the rest. That rule depends on the confidence meaning something.\n\nFine-tuned classifiers are typically overconfident: predictions cluster near 0 and 1, and those at 0.9 are right less often than 90% of the time. Check it by binning held-out predictions by confidence and comparing each bin's mean confidence with its accuracy. Temperature scaling (one scalar fitted on validation data, dividing the logits) corrects most of the gap in about ten lines.\n\n## Knowing when you have enough data\n\nPlot the learning curve. Train on 25%, 50%, 75% and 100% of your labels and plot macro F1 against training size.\n\n- Still climbing steeply at 100%: more labels is your best investment.\n- Flattening: more labels will not help much; better labels, better features or a different taxonomy might.\n\nThis takes four short runs on a 150M model (under an hour on free hardware) and it answers the question teams usually argue about instead: whether to spend the next week labelling.\n\n## Where the encoder stops being right\n\nBe honest about the boundary. An encoder classifier needs labels, cannot take a new class without retraining, and degrades silently when your inputs drift. If your label set changes monthly, or you have fewer than a few hundred examples per class, prompting a general model is the better tool despite costing more per call."},"check":{"question":"A six-class router reports 89% accuracy. The confusion matrix shows the rarest class, 3% of the data, is never predicted at all. Why did accuracy miss this?","options":[{"text":"Accuracy excludes classes with no predictions from its denominator, so the rare class was never counted.","correct":false,"tempting":"Invents an exclusion rule; accuracy counts every item, which is exactly how the rare class's errors get absorbed."},{"text":"Accuracy averages over items, not classes, so failing a 3% class costs three points.","correct":true},{"text":"Accuracy is computed on the training set by default, where the model had memorised the rare class.","correct":false,"tempting":"Attributes the gap to an evaluation-split mistake rather than to what the metric measures, and would show up as a train-test gap instead."},{"text":"The rare class's F1 is undefined when no predictions are made, so it is dropped from the aggregate.","correct":false,"tempting":"Describes a real edge case in macro F1 computation, which is a different metric from the accuracy figure being questioned."}]},"prev":{"slug":"training-a-reranker","title":"Training a reranker","url":"https://addaly.com/learn/fine-tuning/training-a-reranker"},"next":{"slug":"speech-models-and-accents","title":"Fine-tuning a speech model","url":"https://addaly.com/learn/fine-tuning/speech-models-and-accents"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fine-tuning a classifier properly', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/fine-tuning-a-classifier (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fine-tuning a classifier properly","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"A 150M encoder with id2label in its config, a weighted loss or adjusted thresholds for imbalance, macro F1 with the confusion matrix read rather than the summary line, and a four-point learning curve that tells you whether another week of labelling is worth it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-setup","runs":[{"text":"The setup"}]},{"t":"p","runs":[{"text":"Module one argued that a fixed set of labels wants an encoder rather than a generative model. This is how that is done."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoModelForSequenceClassification, AutoTokenizer\n\nMODEL = \"answerdotai/ModernBERT-base\"     # 149M; or microsoft/deberta-v3-base\ntok = AutoTokenizer.from_pretrained(MODEL)\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    MODEL, num_labels=6, id2label=ID2LABEL, label2id=LABEL2ID)"},{"t":"p","runs":[{"text":"Pass "},{"text":"id2label","code":true},{"text":" and "},{"text":"label2id","code":true},{"text":". They are saved in the config, which means the model that comes back off disk in six months says what its outputs mean. Without them you have a tensor of six numbers and a note in somebody’s head."}]},{"t":"p","runs":[{"text":"Full fine-tuning at this size costs about 6 GB of GPU memory at a learning rate of 2e-5 for 3 to 5 epochs. No adapters, no quantisation; the model is small enough that all the machinery from earlier modules is unnecessary."}]},{"t":"h2","id":"class-imbalance-which-you-will-have","runs":[{"text":"Class imbalance, which you will have"}]},{"t":"p","runs":[{"text":"Real label distributions are not uniform. A six-way routing task might be 62% one class and 3% another. Trained naively, the model learns to predict the majority class and scores 62% accuracy while being useless."}]},{"t":"p","runs":[{"text":"Three treatments, in order of preference:"}]},{"t":"p","runs":[{"text":"Weighted loss.","b":true},{"text":" Weight each class inversely to its frequency. One override:"}]},{"t":"code","lang":"python","label":"Python","text":"weights = torch.tensor(len(y) / (n_classes * np.bincount(y)), dtype=torch.float)\nloss = F.cross_entropy(logits, labels, weight=weights.to(logits.device))"},{"t":"p","runs":[{"text":"Threshold adjustment after training.","b":true},{"text":" Train normally, then move the decision boundary per class using a validation set. Often better than reweighting, because it separates “what does the model believe” from “what do we do about it”, and lets you change the second without retraining."}]},{"t":"p","runs":[{"text":"Resampling.","b":true},{"text":" Oversample the minority or undersample the majority. Simple, and oversampling duplicates examples, which invites memorisation of your rarest and most precious class."}]},{"t":"h2","id":"measure-with-macro-f1-not-accuracy","runs":[{"text":"Measure with macro F1, not accuracy"}]},{"t":"p","runs":[{"text":"Accuracy is dominated by the majority class and will tell you a useless model is a good one. Macro-averaged F1 weights every class equally, so the 3% class counts as much as the 62% one."}]},{"t":"p","runs":[{"text":"Print the per-class report, always:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import classification_report, confusion_matrix\nprint(classification_report(y_true, y_pred, target_names=LABELS, digits=3))\nprint(confusion_matrix(y_true, y_pred))"},{"t":"p","runs":[{"text":"Read the confusion matrix rather than the summary line. It tells you "},{"text":"which","i":true},{"text":" classes are being confused with which, and that is almost always actionable: two labels that the model mixes up are usually two labels your annotators mixed up, which is a taxonomy problem rather than a model problem."}]},{"t":"h2","id":"multi-label-is-a-different-loss","runs":[{"text":"Multi-label is a different loss"}]},{"t":"p","runs":[{"text":"If an item can carry several labels at once, the labels are not mutually exclusive and softmax is wrong. Use independent sigmoids:"}]},{"t":"code","lang":"python","label":"Python","text":"model = AutoModelForSequenceClassification.from_pretrained(\n    MODEL, num_labels=12, problem_type=\"multi_label_classification\")"},{"t":"p","runs":[{"text":"That "},{"text":"problem_type","code":true},{"text":" switches the loss to binary cross-entropy per label. Then tune a threshold "},{"text":"per label","b":true},{"text":" on validation data: a single global 0.5 is rarely right when label frequencies differ by an order of magnitude."}]},{"t":"h2","id":"calibration-if-you-will-route-on-confidence","runs":[{"text":"Calibration, if you will route on confidence"}]},{"t":"p","runs":[{"text":"A common and sensible design is to use the classifier for confident cases and fall back to an expensive model for the rest. That rule depends on the confidence meaning something."}]},{"t":"p","runs":[{"text":"Fine-tuned classifiers are typically overconfident: predictions cluster near 0 and 1, and those at 0.9 are right less often than 90% of the time. Check it by binning held-out predictions by confidence and comparing each bin’s mean confidence with its accuracy. Temperature scaling (one scalar fitted on validation data, dividing the logits) corrects most of the gap in about ten lines."}]},{"t":"h2","id":"knowing-when-you-have-enough-data","runs":[{"text":"Knowing when you have enough data"}]},{"t":"p","runs":[{"text":"Plot the learning curve. Train on 25%, 50%, 75% and 100% of your labels and plot macro F1 against training size."}]},{"t":"ul","items":[[{"text":"Still climbing steeply at 100%: more labels is your best investment."}],[{"text":"Flattening: more labels will not help much; better labels, better features or a different taxonomy might."}]]},{"t":"p","runs":[{"text":"This takes four short runs on a 150M model (under an hour on free hardware) and it answers the question teams usually argue about instead: whether to spend the next week labelling."}]},{"t":"h2","id":"where-the-encoder-stops-being-right","runs":[{"text":"Where the encoder stops being right"}]},{"t":"p","runs":[{"text":"Be honest about the boundary. An encoder classifier needs labels, cannot take a new class without retraining, and degrades silently when your inputs drift. If your label set changes monthly, or you have fewer than a few hundred examples per class, prompting a general model is the better tool despite costing more per call."}]}],"check":{"question":[{"text":"A six-class router reports 89% accuracy. The confusion matrix shows the rarest class, 3% of the data, is never predicted at all. Why did accuracy miss this?"}],"options":[{"text":[{"text":"Accuracy excludes classes with no predictions from its denominator, so the rare class was never counted."}],"correct":false,"tempting":[{"text":"Invents an exclusion rule; accuracy counts every item, which is exactly how the rare class’s errors get absorbed."}]},{"text":[{"text":"Accuracy averages over items, not classes, so failing a 3% class costs three points."}],"correct":true,"tempting":null},{"text":[{"text":"Accuracy is computed on the training set by default, where the model had memorised the rare class."}],"correct":false,"tempting":[{"text":"Attributes the gap to an evaluation-split mistake rather than to what the metric measures, and would show up as a train-test gap instead."}]},{"text":[{"text":"The rare class’s F1 is undefined when no predictions are made, so it is dropped from the aggregate."}],"correct":false,"tempting":[{"text":"Describes a real edge case in macro F1 computation, which is a different metric from the accuracy figure being questioned."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"training-a-reranker","title":"Training a reranker","minutes":8},"next":{"slug":"speech-models-and-accents","title":"Fine-tuning a speech model","minutes":9}}}