Teaching it which answer is better

The gap supervised tuning leaves

Supervised fine-tuning shows the model one good answer per input. It never sees a bad one, so it learns what a good answer looks like without learning what to avoid.

That leaves a specific set of problems unsolved: the model is too verbose, or too terse; it hedges when it should commit; it adds a summary paragraph nobody wanted; it refuses things it should handle and handles things it should refuse; it picks the second-best of two acceptable formats. These are preferences between outputs, and you often cannot write down the rule: you only know it when you see two answers side by side.

Preference tuning learns from exactly that: pairs of answers with a judgement about which is better.

The three methods you will meet

DPO: direct preference optimisation. Takes triples of prompt, chosen answer and rejected answer, and adjusts the model to raise the relative likelihood of the chosen one. It needs a frozen reference copy of the starting model, and a parameter beta controls how far the tuned model is allowed to drift from it. Lower beta means more change and more risk.

ORPO. Combines supervised and preference learning in a single stage with no reference model, so it is cheaper in memory and simpler to run. Useful when you are starting from a base model and would otherwise do two runs.

KTO. Needs only a binary judgement (this output was good, that one was bad) rather than matched pairs. This matters enormously in practice, because binary feedback is what real products collect. A thumbs-up button produces KTO data; producing DPO pairs requires someone to compare two answers to the same input deliberately.

from trl import DPOTrainer, DPOConfig
trainer = DPOTrainer(
    model=sft_model, ref_model=None,          # None uses the adapter-disabled base
    args=DPOConfig(beta=0.1, learning_rate=5e-6, num_train_epochs=1,
                   per_device_train_batch_size=1, gradient_accumulation_steps=16),
    train_dataset=pairs,                       # prompt / chosen / rejected
    tokenizer=tok, peft_config=lora_cfg,
)

Note the learning rate: preference tuning wants roughly an order of magnitude lower than supervised tuning, around 5e-6 to 5e-7. Using an SFT learning rate here is the standard way to destroy a model in twenty minutes.

Where the pairs come from

From your own model. Sample two completions at temperature 0.8 for each of 500 real prompts, and have a person pick the better one. Fast (a judgement takes seconds) and the pairs are on-distribution, which matters: preference data drawn from a different model teaches your model less.

From corrections. The original output is the rejected answer; the human-edited version is the chosen one. This is free if you are already correcting output, and it is the highest-quality source available.

From a judge model, with a rubric, spot-checked by a person. Cheap and biased in ways the judge is biased, so audit twenty judgements before trusting the rest.

Five hundred to two thousand pairs is the usual range. As with supervised data, consistency matters more than volume; pairs judged by three people with different tastes teach the average of three tastes, which is nobody’s.

The order

Supervised tuning first, preference tuning second. Preference tuning adjusts a model that already does the task; it is a poor way to teach the task itself. Running DPO on a base model that has never seen your format mostly produces confusion.

The damage to watch for

Preference tuning is the stage most likely to make a model worse in ways your task metric will not show.

• Verbosity drift. Human raters, and judge models even more, prefer longer answers. Uncontrolled preference tuning reliably makes models more verbose. If your data was rated by a model, check length distributions before and after.
• Capability loss. Too many epochs or too low a beta pulls the model far from the reference and general ability degrades. One epoch is usually correct.
• Reward hacking on format. The model learns the surface features that correlated with being chosen (headings, bullet points, a confident opening) rather than the substance.

The defence is the same in every case: a regression set covering general capability, other languages and refusal behaviour, run before and after. The next lesson is about building one, and preference tuning is the stage that most needs it.