What supervised fine-tuning cannot express

One gold answer, maximised

Supervised fine-tuning maximises the probability of a single written answer given an input. That is its whole mechanism, and four limitations follow directly from it.

1. There is no signal about what is bad. The loss says “make this more likely”. It never says “make that less likely”. If your model has a failure mode (a hedge you dislike, a formatting habit, a tendency to answer questions it should decline) SFT can only address it by showing many examples of the good behaviour and hoping the bad one is crowded out. You cannot point at the failure.

2. All gold answers are equally gold. An example your best annotator wrote carefully and an example somebody rushed at 4 p.m. contribute identical gradients. SFT has no way to represent “this one is excellent, that one is merely acceptable”. The model learns the average of your dataset’s quality, whatever that is.

3. It is off-policy. The model is trained on text it did not produce, which means training never visits the places the model actually goes wrong. Its own characteristic errors are absent from the data, so nothing corrects them.

4. Exposure bias. At training time every token is conditioned on a correct prefix. At generation time the model conditions on its own output, including its own mistakes, and it has never been shown what to do after a mistake.

The decision rule for the whole module

These limitations sort cleanly onto three methods, and the sorting question is: what can you actually produce?

• You can write the right answer. Use SFT. Format, style, schema adherence, domain register, tool-call syntax: anything where a competent person can type the correct output. This is the large majority of applied fine-tuning, and the rest of this module is not for you.
• You cannot write the right answer, but you can pick the better of two. Use a preference method. Helpfulness, tone, how much to hedge, when to ask a clarifying question, how long an answer should be. Judging is far easier than producing, and preference methods are built on exactly that asymmetry.
• You can check an answer with a program. Use a verifiable reward. Maths with a known result, code that must pass tests, SQL that must return the right rows, JSON that must match a schema. Here you have something stronger than a preference: you have ground truth, cheaply and at scale.

Most teams reach for preference methods when they are in the first row and would have been better served by writing fifty more examples.

What preference data adds, mechanically

A preference pair is one input with two responses and a label saying which is better. The training signal is relative: raise the probability of the chosen response, lower the probability of the rejected one, and do so relative to where the model started.

That relative framing is what SFT lacks. It lets you express things that have no single correct form (“shorter is better here”, “do not apologise before answering”, “ask rather than guess when the order number is missing”) as a contrast rather than as a demonstration.

And crucially, the rejected response can be the model’s own output. That fixes the off-policy problem: you are training on the model’s actual failures rather than on a curated corpus in which they do not occur.

The order of operations

Preference methods assume a model that is already roughly right. They are a refinement step, not a replacement for SFT, and applying them to a base model that has never been instruction-tuned produces disappointing results, because there is nothing coherent to refine.

The standard pipeline is:

1. SFT on demonstrations, to establish the format and the basic behaviour.
2. Preference or reward optimisation on top, to refine the qualities you can judge but not write.

ORPO, covered two lessons on, is the notable exception that merges these into one stage.

The honest scale check

Preference tuning is where the effort curve steepens sharply. You need a trained-and-working SFT model first, a preference dataset (thousands of pairs, each requiring a judgement), reference log-probabilities from the starting model (by default a second copy held in memory, though they can be precomputed once before training, or taken from the base with a LoRA adapter switched off; ORPO and SimPO need none), and an evaluation that can detect changes too subtle to write down, because if you could write them down, you would have written a demonstration instead.

Before committing to that, answer one question honestly: can I state what I want in a sentence a person could follow? If yes, write twenty demonstrations of it and try SFT. The number of teams who reach for DPO and discover afterwards that their real problem was a chat template bug is not small.