{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"heads-instead-of-tokens","title":"Training a head instead of generating tokens","url":"https://addaly.com/learn/fine-tuning/heads-instead-of-tokens","position":47,"of":84,"minutes":9,"answer":"Swapping the token head for a linear head turns a decoder into a scorer that costs one forward pass instead of a generation loop.","keyIdea":"Swapping the token head for a linear head turns a decoder into a scorer that costs one forward pass instead of a generation loop, but a preference-trained reward model learns an ordering with no absolute scale, and a decoder classifier pools the last non-padding token, so padding side and pad_token_id must both be set."},"body":{"format":"markdown","text":"## Replace the last layer\n\nA decoder's final layer projects a hidden state to vocabulary size, producing a distribution over the next token. Replace that projection with a small linear layer producing one number, or six, and you have a scorer or a classifier built on the same pretrained body.\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen/Qwen2.5-1.5B\", num_labels=1)   # 1 = regression / reward\n```\n\nThis is what a reward model is. It is also what a quality scorer, a safety classifier, a routing model and a reranker are. In every case you have traded the ability to write text for a single cheap forward pass: no autoregressive decoding, no sampling, a fixed and tiny output.\n\n## Why the cost difference is large\n\nGenerating a 200-token judgement takes 200 sequential forward passes through the model. Scoring with a head takes one. At serving time that is roughly two orders of magnitude of difference in latency and cost, for a task whose output was always going to be a number.\n\nIf you are currently prompting a model to \"rate this response from 1 to 10\" and parsing the digit out of its reply, you are paying for a generation loop to produce one token, and a head would do it better.\n\n## The padding bug\n\nHere is the mistake that costs people a day. For a decoder-based classifier, the pooled representation is taken from the **last non-padding token**. If your tokenizer pads on the right, the final position of a short sequence is a pad token, and naive pooling reads the hidden state of padding.\n\nTwo defences, and you want both:\n\n```python\ntok.padding_side = \"left\"        # for decoder-based classification\nmodel.config.pad_token_id = tok.pad_token_id   # so pooling can find the end\n```\n\nIf `pad_token_id` is unset, and for several base models it is, `AutoModelForSequenceClassification` cannot locate the last real token and will either raise or quietly use the wrong position. Encoder models such as BERT and ModernBERT use a dedicated classification token and do not have this problem, which is one more reason to prefer an encoder when the task is classification.\n\n## Losses, by output type\n\n- **Binary classification**: binary cross-entropy, one output.\n- **Multi-class**: cross-entropy, n outputs.\n- **Multi-label**: binary cross-entropy per label, n independent sigmoids. Do not use softmax here; the labels are not mutually exclusive.\n- **Regression to a human score**: mean squared error, one output. Standardise the targets first.\n- **Reward model from preferences**: the Bradley-Terry loss. Score both responses, maximise the probability that the chosen scores higher:\n\n```python\nloss = -F.logsigmoid(score_chosen - score_rejected).mean()\n```\n\nThat last one is worth noticing: a reward model never learns an absolute scale. It learns an ordering. A score of 3.2 means nothing on its own: only that it is above or below another response's score. Teams that build a reward model and then set a fixed threshold on its output are assuming a calibration that was never trained.\n\n## Calibration\n\nA classifier's sigmoid output is not a probability until you have checked that it is. Fine-tuned models are routinely overconfident: the outputs cluster near 0 and 1, and the ones at 0.9 are correct rather less than 90% of the time.\n\nMeasure it before you rely on it. Bin your held-out predictions by confidence and compare each bin's mean confidence to its accuracy. If the gap is large, temperature scaling (a single scalar fitted on a validation set, dividing the logits before the sigmoid) fixes most of it for the cost of ten lines. This matters specifically when you are using the confidence to route: \"fall back to the expensive model below 0.7\" is a rule about a number that has to mean something.\n\n## Where this belongs in a fine-tuning project\n\nThree places, and all three are common:\n\n1. **The reward model**, if you are doing PPO.\n2. **An automatic evaluator.** Train a small scorer on a few thousand human judgements of your own outputs, and you have an evaluation you can run on every checkpoint for free. Check its agreement with held-out human labels before trusting it, and re-check when the model changes.\n3. **A router or a safety gate** in front of a generative model, deciding which requests go to the cheap path, the expensive path, or a refusal.\n\nAll three are small models, quick to train, cheap to serve, and easy to evaluate properly, which makes them the most under-used tool in the applied fine-tuning kit."},"check":{"question":"A team trains a reward model on preference pairs with the Bradley-Terry loss, then deploys it with a rule that any response scoring below 2.0 is rejected. Why is this rule unsound?","options":[{"text":"Reward models output logits rather than scores, so 2.0 has to be passed through a sigmoid before it can be compared.","correct":false,"tempting":"Suggests a transformation would fix it, when applying a sigmoid to an uncalibrated ordering still yields no meaningful absolute threshold."},{"text":"Reward models drift during training, so any fixed threshold becomes stale and must be recomputed each epoch.","correct":false,"tempting":"Identifies a maintenance issue rather than the structural one; the threshold would be unsound even on a fully converged model."},{"text":"The loss trains only the ordering, so a single score has no calibrated absolute meaning.","correct":true},{"text":"Scores depend on response length, so the threshold rejects short responses regardless of quality.","correct":false,"tempting":"Names a real bias in reward models, but it is a separate problem from the score having no absolute scale at all."}]},"prev":{"slug":"continued-pretraining","title":"Continued pretraining on raw text","url":"https://addaly.com/learn/fine-tuning/continued-pretraining"},"next":{"slug":"templates-and-masking","title":"Chat Templates and Loss Masking","url":"https://addaly.com/learn/fine-tuning/templates-and-masking"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Training a head instead of generating tokens', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/heads-instead-of-tokens (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Training a head instead of generating tokens","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"Swapping the token head for a linear head turns a decoder into a scorer that costs one forward pass instead of a generation loop, but a preference-trained reward model learns an ordering with no absolute scale, and a decoder classifier pools the last non-padding token, so padding side and pad_token_id must both be set.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"replace-the-last-layer","runs":[{"text":"Replace the last layer"}]},{"t":"p","runs":[{"text":"A decoder’s final layer projects a hidden state to vocabulary size, producing a distribution over the next token. Replace that projection with a small linear layer producing one number, or six, and you have a scorer or a classifier built on the same pretrained body."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoModelForSequenceClassification\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen/Qwen2.5-1.5B\", num_labels=1)   # 1 = regression / reward"},{"t":"p","runs":[{"text":"This is what a reward model is. It is also what a quality scorer, a safety classifier, a routing model and a reranker are. In every case you have traded the ability to write text for a single cheap forward pass: no autoregressive decoding, no sampling, a fixed and tiny output."}]},{"t":"h2","id":"why-the-cost-difference-is-large","runs":[{"text":"Why the cost difference is large"}]},{"t":"p","runs":[{"text":"Generating a 200-token judgement takes 200 sequential forward passes through the model. Scoring with a head takes one. At serving time that is roughly two orders of magnitude of difference in latency and cost, for a task whose output was always going to be a number."}]},{"t":"p","runs":[{"text":"If you are currently prompting a model to “rate this response from 1 to 10” and parsing the digit out of its reply, you are paying for a generation loop to produce one token, and a head would do it better."}]},{"t":"h2","id":"the-padding-bug","runs":[{"text":"The padding bug"}]},{"t":"p","runs":[{"text":"Here is the mistake that costs people a day. For a decoder-based classifier, the pooled representation is taken from the "},{"text":"last non-padding token","b":true},{"text":". If your tokenizer pads on the right, the final position of a short sequence is a pad token, and naive pooling reads the hidden state of padding."}]},{"t":"p","runs":[{"text":"Two defences, and you want both:"}]},{"t":"code","lang":"python","label":"Python","text":"tok.padding_side = \"left\"        # for decoder-based classification\nmodel.config.pad_token_id = tok.pad_token_id   # so pooling can find the end"},{"t":"p","runs":[{"text":"If "},{"text":"pad_token_id","code":true},{"text":" is unset, and for several base models it is, "},{"text":"AutoModelForSequenceClassification","code":true},{"text":" cannot locate the last real token and will either raise or quietly use the wrong position. Encoder models such as BERT and ModernBERT use a dedicated classification token and do not have this problem, which is one more reason to prefer an encoder when the task is classification."}]},{"t":"h2","id":"losses-by-output-type","runs":[{"text":"Losses, by output type"}]},{"t":"ul","items":[[{"text":"Binary classification","b":true},{"text":": binary cross-entropy, one output."}],[{"text":"Multi-class","b":true},{"text":": cross-entropy, n outputs."}],[{"text":"Multi-label","b":true},{"text":": binary cross-entropy per label, n independent sigmoids. Do not use softmax here; the labels are not mutually exclusive."}],[{"text":"Regression to a human score","b":true},{"text":": mean squared error, one output. Standardise the targets first."}],[{"text":"Reward model from preferences","b":true},{"text":": the Bradley-Terry loss. Score both responses, maximise the probability that the chosen scores higher:"}]]},{"t":"code","lang":"python","label":"Python","text":"loss = -F.logsigmoid(score_chosen - score_rejected).mean()"},{"t":"p","runs":[{"text":"That last one is worth noticing: a reward model never learns an absolute scale. It learns an ordering. A score of 3.2 means nothing on its own: only that it is above or below another response’s score. Teams that build a reward model and then set a fixed threshold on its output are assuming a calibration that was never trained."}]},{"t":"h2","id":"calibration","runs":[{"text":"Calibration"}]},{"t":"p","runs":[{"text":"A classifier’s sigmoid output is not a probability until you have checked that it is. Fine-tuned models are routinely overconfident: the outputs cluster near 0 and 1, and the ones at 0.9 are correct rather less than 90% of the time."}]},{"t":"p","runs":[{"text":"Measure it before you rely on it. Bin your held-out predictions by confidence and compare each bin’s mean confidence to its accuracy. If the gap is large, temperature scaling (a single scalar fitted on a validation set, dividing the logits before the sigmoid) fixes most of it for the cost of ten lines. This matters specifically when you are using the confidence to route: “fall back to the expensive model below 0.7” is a rule about a number that has to mean something."}]},{"t":"h2","id":"where-this-belongs-in-a-fine-tuning-project","runs":[{"text":"Where this belongs in a fine-tuning project"}]},{"t":"p","runs":[{"text":"Three places, and all three are common:"}]},{"t":"ol","start":1,"items":[[{"text":"The reward model","b":true},{"text":", if you are doing PPO."}],[{"text":"An automatic evaluator.","b":true},{"text":" Train a small scorer on a few thousand human judgements of your own outputs, and you have an evaluation you can run on every checkpoint for free. Check its agreement with held-out human labels before trusting it, and re-check when the model changes."}],[{"text":"A router or a safety gate","b":true},{"text":" in front of a generative model, deciding which requests go to the cheap path, the expensive path, or a refusal."}]]},{"t":"p","runs":[{"text":"All three are small models, quick to train, cheap to serve, and easy to evaluate properly, which makes them the most under-used tool in the applied fine-tuning kit."}]}],"check":{"question":[{"text":"A team trains a reward model on preference pairs with the Bradley-Terry loss, then deploys it with a rule that any response scoring below 2.0 is rejected. Why is this rule unsound?"}],"options":[{"text":[{"text":"Reward models output logits rather than scores, so 2.0 has to be passed through a sigmoid before it can be compared."}],"correct":false,"tempting":[{"text":"Suggests a transformation would fix it, when applying a sigmoid to an uncalibrated ordering still yields no meaningful absolute threshold."}]},{"text":[{"text":"Reward models drift during training, so any fixed threshold becomes stale and must be recomputed each epoch."}],"correct":false,"tempting":[{"text":"Identifies a maintenance issue rather than the structural one; the threshold would be unsound even on a fully converged model."}]},{"text":[{"text":"The loss trains only the ordering, so a single score has no calibrated absolute meaning."}],"correct":true,"tempting":null},{"text":[{"text":"Scores depend on response length, so the threshold rejects short responses regardless of quality."}],"correct":false,"tempting":[{"text":"Names a real bias in reward models, but it is a separate problem from the score having no absolute scale at all."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"continued-pretraining","title":"Continued pretraining on raw text","minutes":9},"next":{"slug":"templates-and-masking","title":"Chat Templates and Loss Masking","minutes":9}}}