{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"sft-is-imitation","title":"What supervised fine-tuning cannot express","url":"https://addaly.com/learn/fine-tuning/sft-is-imitation","position":39,"of":84,"minutes":9,"answer":"SFT can only say 'make this more likely', so it cannot express that one answer is better than another or correct a failure it never sees.","keyIdea":"SFT can only say 'make this more likely', so it cannot express that one answer is better than another or correct a failure it never sees, which is why the method you need is decided by whether you can write the right answer, only rank two answers, or check one with a program."},"body":{"format":"markdown","text":"## One gold answer, maximised\n\nSupervised fine-tuning maximises the probability of a single written answer given an input. That is its whole mechanism, and four limitations follow directly from it.\n\n**1. There is no signal about what is bad.** The loss says \"make this more likely\". It never says \"make that less likely\". If your model has a failure mode (a hedge you dislike, a formatting habit, a tendency to answer questions it should decline) SFT can only address it by showing many examples of the good behaviour and hoping the bad one is crowded out. You cannot point at the failure.\n\n**2. All gold answers are equally gold.** An example your best annotator wrote carefully and an example somebody rushed at 4 p.m. contribute identical gradients. SFT has no way to represent \"this one is excellent, that one is merely acceptable\". The model learns the average of your dataset's quality, whatever that is.\n\n**3. It is off-policy.** The model is trained on text it did not produce, which means training never visits the places the model actually goes wrong. Its own characteristic errors are absent from the data, so nothing corrects them.\n\n**4. Exposure bias.** At training time every token is conditioned on a correct prefix. At generation time the model conditions on its own output, including its own mistakes, and it has never been shown what to do after a mistake.\n\n## The decision rule for the whole module\n\nThese limitations sort cleanly onto three methods, and the sorting question is: **what can you actually produce?**\n\n- **You can write the right answer.** Use SFT. Format, style, schema adherence, domain register, tool-call syntax: anything where a competent person can type the correct output. This is the large majority of applied fine-tuning, and the rest of this module is not for you.\n- **You cannot write the right answer, but you can pick the better of two.** Use a preference method. Helpfulness, tone, how much to hedge, when to ask a clarifying question, how long an answer should be. Judging is far easier than producing, and preference methods are built on exactly that asymmetry.\n- **You can check an answer with a program.** Use a verifiable reward. Maths with a known result, code that must pass tests, SQL that must return the right rows, JSON that must match a schema. Here you have something stronger than a preference: you have ground truth, cheaply and at scale.\n\nMost teams reach for preference methods when they are in the first row and would have been better served by writing fifty more examples.\n\n## What preference data adds, mechanically\n\nA preference pair is one input with two responses and a label saying which is better. The training signal is *relative*: raise the probability of the chosen response, lower the probability of the rejected one, and do so relative to where the model started.\n\nThat relative framing is what SFT lacks. It lets you express things that have no single correct form (\"shorter is better here\", \"do not apologise before answering\", \"ask rather than guess when the order number is missing\") as a contrast rather than as a demonstration.\n\n> Figure: What can you actually produce?. Steps: 1. You can write the right answer (Supervised fine-tuning. Format, style, schema adherence, domain register, tool-call syntax: anything a competent person can type.); 2. You can only pick the better of two (A preference method. Helpfulness, tone, how much to hedge, when to ask rather than guess. Judging is far easier than producing.); 3. A program can check the answer (A verifiable reward. Maths with a known result, code that must pass tests, SQL that must return the right rows, JSON that must validate.). The sorting question for the whole module. Most teams reach for a preference method while sitting in the first row, and would have been better served by writing fifty more demonstrations.\n\nAnd crucially, the rejected response can be *the model's own output*. That fixes the off-policy problem: you are training on the model's actual failures rather than on a curated corpus in which they do not occur.\n\n## The order of operations\n\nPreference methods assume a model that is already roughly right. They are a refinement step, not a replacement for SFT, and applying them to a base model that has never been instruction-tuned produces disappointing results, because there is nothing coherent to refine.\n\nThe standard pipeline is:\n\n1. **SFT** on demonstrations, to establish the format and the basic behaviour.\n2. **Preference or reward optimisation** on top, to refine the qualities you can judge but not write.\n\nORPO, covered two lessons on, is the notable exception that merges these into one stage.\n\n## The honest scale check\n\nPreference tuning is where the effort curve steepens sharply. You need a trained-and-working SFT model first, a preference dataset (thousands of pairs, each requiring a judgement), reference log-probabilities from the starting model (by default a second copy held in memory, though they can be precomputed once before training, or taken from the base with a LoRA adapter switched off; ORPO and SimPO need none), and an evaluation that can detect changes too subtle to write down, because if you could write them down, you would have written a demonstration instead.\n\nBefore committing to that, answer one question honestly: *can I state what I want in a sentence a person could follow?* If yes, write twenty demonstrations of it and try SFT. The number of teams who reach for DPO and discover afterwards that their real problem was a chat template bug is not small."},"check":{"question":"A model occasionally opens answers with an apology the team dislikes. They add 500 SFT examples with no apologies and the habit persists at a reduced rate. Why does SFT struggle with this specifically?","options":[{"text":"The apology is in the base model's pretraining data, and fine-tuning cannot alter pretrained behaviour.","correct":false,"tempting":"Overstates the limit; fine-tuning routinely changes pretrained habits, just not efficiently through demonstration alone."},{"text":"Five hundred examples is too few relative to the base model's size for any behavioural change to take hold.","correct":false,"tempting":"Treats it as a volume problem, when the same 500 examples do change other behaviours in the same run."},{"text":"The loss can only raise the probability of what is shown, never lower the apology's, so it is not targetable.","correct":true},{"text":"Apologies appear at the start of a response, where the loss mask most often excludes tokens, so they receive little gradient.","correct":false,"tempting":"Invents a positional gap in masking; completion-only masking covers the whole assistant turn including its first tokens."}]},"prev":{"slug":"documenting-the-dataset","title":"The data card","url":"https://addaly.com/learn/fine-tuning/documenting-the-dataset"},"next":{"slug":"where-preference-data-comes-from","title":"Where preference data comes from","url":"https://addaly.com/learn/fine-tuning/where-preference-data-comes-from"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What supervised fine-tuning cannot express', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/sft-is-imitation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What supervised fine-tuning cannot express","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"SFT can only say ‘make this more likely’, so it cannot express that one answer is better than another or correct a failure it never sees, which is why the method you need is decided by whether you can write the right answer, only rank two answers, or check one with a program.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 5.1","figure":{"type":"flow","steps":[{"label":"You can write the right answer","detail":"Supervised fine-tuning. Format, style, schema adherence, domain register, tool-call syntax: anything a competent person can type."},{"label":"You can only pick the better of two","detail":"A preference method. Helpfulness, tone, how much to hedge, when to ask rather than guess. Judging is far easier than producing."},{"label":"A program can check the answer","detail":"A verifiable reward. Maths with a known result, code that must pass tests, SQL that must return the right rows, JSON that must validate."}],"title":"What can you actually produce?","caption":"The sorting question for the whole module. Most teams reach for a preference method while sitting in the first row, and would have been better served by writing fifty more demonstrations."},"alt":"Figure: What can you actually produce?. Steps: 1. You can write the right answer (Supervised fine-tuning. Format, style, schema adherence, domain register, tool-call syntax: anything a competent person can type.); 2. You can only pick the better of two (A preference method. Helpfulness, tone, how much to hedge, when to ask rather than guess. Judging is far easier than producing.); 3. A program can check the answer (A verifiable reward. Maths with a known result, code that must pass tests, SQL that must return the right rows, JSON that must validate.). The sorting question for the whole module. Most teams reach for a preference method while sitting in the first row, and would have been better served by writing fifty more demonstrations."}},"blocks":[{"t":"h2","id":"one-gold-answer-maximised","runs":[{"text":"One gold answer, maximised"}]},{"t":"p","runs":[{"text":"Supervised fine-tuning maximises the probability of a single written answer given an input. That is its whole mechanism, and four limitations follow directly from it."}]},{"t":"p","runs":[{"text":"1. There is no signal about what is bad.","b":true},{"text":" The loss says “make this more likely”. It never says “make that less likely”. If your model has a failure mode (a hedge you dislike, a formatting habit, a tendency to answer questions it should decline) SFT can only address it by showing many examples of the good behaviour and hoping the bad one is crowded out. You cannot point at the failure."}]},{"t":"p","runs":[{"text":"2. All gold answers are equally gold.","b":true},{"text":" An example your best annotator wrote carefully and an example somebody rushed at 4 p.m. contribute identical gradients. SFT has no way to represent “this one is excellent, that one is merely acceptable”. The model learns the average of your dataset’s quality, whatever that is."}]},{"t":"p","runs":[{"text":"3. It is off-policy.","b":true},{"text":" The model is trained on text it did not produce, which means training never visits the places the model actually goes wrong. Its own characteristic errors are absent from the data, so nothing corrects them."}]},{"t":"p","runs":[{"text":"4. Exposure bias.","b":true},{"text":" At training time every token is conditioned on a correct prefix. At generation time the model conditions on its own output, including its own mistakes, and it has never been shown what to do after a mistake."}]},{"t":"h2","id":"the-decision-rule-for-the-whole-module","runs":[{"text":"The decision rule for the whole module"}]},{"t":"p","runs":[{"text":"These limitations sort cleanly onto three methods, and the sorting question is: "},{"text":"what can you actually produce?","b":true}]},{"t":"ul","items":[[{"text":"You can write the right answer.","b":true},{"text":" Use SFT. Format, style, schema adherence, domain register, tool-call syntax: anything where a competent person can type the correct output. This is the large majority of applied fine-tuning, and the rest of this module is not for you."}],[{"text":"You cannot write the right answer, but you can pick the better of two.","b":true},{"text":" Use a preference method. Helpfulness, tone, how much to hedge, when to ask a clarifying question, how long an answer should be. Judging is far easier than producing, and preference methods are built on exactly that asymmetry."}],[{"text":"You can check an answer with a program.","b":true},{"text":" Use a verifiable reward. Maths with a known result, code that must pass tests, SQL that must return the right rows, JSON that must match a schema. Here you have something stronger than a preference: you have ground truth, cheaply and at scale."}]]},{"t":"p","runs":[{"text":"Most teams reach for preference methods when they are in the first row and would have been better served by writing fifty more examples."}]},{"t":"h2","id":"what-preference-data-adds-mechanically","runs":[{"text":"What preference data adds, mechanically"}]},{"t":"p","runs":[{"text":"A preference pair is one input with two responses and a label saying which is better. The training signal is "},{"text":"relative","i":true},{"text":": raise the probability of the chosen response, lower the probability of the rejected one, and do so relative to where the model started."}]},{"t":"p","runs":[{"text":"That relative framing is what SFT lacks. It lets you express things that have no single correct form (“shorter is better here”, “do not apologise before answering”, “ask rather than guess when the order number is missing”) as a contrast rather than as a demonstration."}]},{"t":"p","runs":[{"text":"And crucially, the rejected response can be "},{"text":"the model’s own output","i":true},{"text":". That fixes the off-policy problem: you are training on the model’s actual failures rather than on a curated corpus in which they do not occur."}]},{"t":"h2","id":"the-order-of-operations","runs":[{"text":"The order of operations"}]},{"t":"p","runs":[{"text":"Preference methods assume a model that is already roughly right. They are a refinement step, not a replacement for SFT, and applying them to a base model that has never been instruction-tuned produces disappointing results, because there is nothing coherent to refine."}]},{"t":"p","runs":[{"text":"The standard pipeline is:"}]},{"t":"ol","start":1,"items":[[{"text":"SFT","b":true},{"text":" on demonstrations, to establish the format and the basic behaviour."}],[{"text":"Preference or reward optimisation","b":true},{"text":" on top, to refine the qualities you can judge but not write."}]]},{"t":"p","runs":[{"text":"ORPO, covered two lessons on, is the notable exception that merges these into one stage."}]},{"t":"h2","id":"the-honest-scale-check","runs":[{"text":"The honest scale check"}]},{"t":"p","runs":[{"text":"Preference tuning is where the effort curve steepens sharply. You need a trained-and-working SFT model first, a preference dataset (thousands of pairs, each requiring a judgement), reference log-probabilities from the starting model (by default a second copy held in memory, though they can be precomputed once before training, or taken from the base with a LoRA adapter switched off; ORPO and SimPO need none), and an evaluation that can detect changes too subtle to write down, because if you could write them down, you would have written a demonstration instead."}]},{"t":"p","runs":[{"text":"Before committing to that, answer one question honestly: "},{"text":"can I state what I want in a sentence a person could follow?","i":true},{"text":" If yes, write twenty demonstrations of it and try SFT. The number of teams who reach for DPO and discover afterwards that their real problem was a chat template bug is not small."}]}],"check":{"question":[{"text":"A model occasionally opens answers with an apology the team dislikes. They add 500 SFT examples with no apologies and the habit persists at a reduced rate. Why does SFT struggle with this specifically?"}],"options":[{"text":[{"text":"The apology is in the base model’s pretraining data, and fine-tuning cannot alter pretrained behaviour."}],"correct":false,"tempting":[{"text":"Overstates the limit; fine-tuning routinely changes pretrained habits, just not efficiently through demonstration alone."}]},{"text":[{"text":"Five hundred examples is too few relative to the base model’s size for any behavioural change to take hold."}],"correct":false,"tempting":[{"text":"Treats it as a volume problem, when the same 500 examples do change other behaviours in the same run."}]},{"text":[{"text":"The loss can only raise the probability of what is shown, never lower the apology’s, so it is not targetable."}],"correct":true,"tempting":null},{"text":[{"text":"Apologies appear at the start of a response, where the loss mask most often excludes tokens, so they receive little gradient."}],"correct":false,"tempting":[{"text":"Invents a positional gap in masking; completion-only masking covers the whole assistant turn including its first tokens."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"documenting-the-dataset","title":"The data card","minutes":8},"next":{"slug":"where-preference-data-comes-from","title":"Where preference data comes from","minutes":9}}}