Training a head instead of generating tokens

Replace the last layer

A decoder’s final layer projects a hidden state to vocabulary size, producing a distribution over the next token. Replace that projection with a small linear layer producing one number, or six, and you have a scorer or a classifier built on the same pretrained body.

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen/Qwen2.5-1.5B", num_labels=1)   # 1 = regression / reward

This is what a reward model is. It is also what a quality scorer, a safety classifier, a routing model and a reranker are. In every case you have traded the ability to write text for a single cheap forward pass: no autoregressive decoding, no sampling, a fixed and tiny output.

Why the cost difference is large

Generating a 200-token judgement takes 200 sequential forward passes through the model. Scoring with a head takes one. At serving time that is roughly two orders of magnitude of difference in latency and cost, for a task whose output was always going to be a number.

If you are currently prompting a model to “rate this response from 1 to 10” and parsing the digit out of its reply, you are paying for a generation loop to produce one token, and a head would do it better.

The padding bug

Here is the mistake that costs people a day. For a decoder-based classifier, the pooled representation is taken from the last non-padding token. If your tokenizer pads on the right, the final position of a short sequence is a pad token, and naive pooling reads the hidden state of padding.

Two defences, and you want both:

tok.padding_side = "left"        # for decoder-based classification
model.config.pad_token_id = tok.pad_token_id   # so pooling can find the end

If pad_token_id is unset, and for several base models it is, AutoModelForSequenceClassification cannot locate the last real token and will either raise or quietly use the wrong position. Encoder models such as BERT and ModernBERT use a dedicated classification token and do not have this problem, which is one more reason to prefer an encoder when the task is classification.

Losses, by output type

• Binary classification: binary cross-entropy, one output.
• Multi-class: cross-entropy, n outputs.
• Multi-label: binary cross-entropy per label, n independent sigmoids. Do not use softmax here; the labels are not mutually exclusive.
• Regression to a human score: mean squared error, one output. Standardise the targets first.
• Reward model from preferences: the Bradley-Terry loss. Score both responses, maximise the probability that the chosen scores higher:

loss = -F.logsigmoid(score_chosen - score_rejected).mean()

That last one is worth noticing: a reward model never learns an absolute scale. It learns an ordering. A score of 3.2 means nothing on its own: only that it is above or below another response’s score. Teams that build a reward model and then set a fixed threshold on its output are assuming a calibration that was never trained.

Calibration

A classifier’s sigmoid output is not a probability until you have checked that it is. Fine-tuned models are routinely overconfident: the outputs cluster near 0 and 1, and the ones at 0.9 are correct rather less than 90% of the time.

Measure it before you rely on it. Bin your held-out predictions by confidence and compare each bin’s mean confidence to its accuracy. If the gap is large, temperature scaling (a single scalar fitted on a validation set, dividing the logits before the sigmoid) fixes most of it for the cost of ten lines. This matters specifically when you are using the confidence to route: “fall back to the expensive model below 0.7” is a rule about a number that has to mean something.

Where this belongs in a fine-tuning project

Three places, and all three are common:

1. The reward model, if you are doing PPO.
2. An automatic evaluator. Train a small scorer on a few thousand human judgements of your own outputs, and you have an evaluation you can run on every checkpoint for free. Check its agreement with held-out human labels before trusting it, and re-check when the model changes.
3. A router or a safety gate in front of a generative model, deciding which requests go to the cheap path, the expensive path, or a refusal.

All three are small models, quick to train, cheap to serve, and easy to evaluate properly, which makes them the most under-used tool in the applied fine-tuning kit.