Pretraining, fine-tuning, and preference training

Three stages that do different jobs

The model you talk to was built in stages. Confusing them is the source of a great many expensive mistakes.

Stage one: pretraining

Next-token prediction over an enormous corpus: web text, books, code, forums, in many languages. Months of compute on thousands of accelerators. This is the stage that costs tens of millions of dollars and produces essentially all of the model’s knowledge and capability.

What comes out is a document continuer. Ask a raw pretrained model “What is the capital of Peru?” and a perfectly reasonable output is a list of five more quiz questions, because that is what such a line is usually followed by in the wild. It knows a great deal and has no idea it is supposed to answer you.

One consequence to keep in mind: the knowledge cut-off belongs to this stage. Later stages do not add news.

Stage two: supervised fine-tuning

Continue training on a much smaller, curated set of examples: typically tens of thousands of prompt-and-response pairs written or vetted by people. Hours or days of compute, not months.

This is where a document continuer becomes an assistant. It learns the shape of the interaction: a question is followed by an answer, a request for code is followed by code, a chat turn ends rather than rambling into an invented reply from the user.

Stage three: preference training

Show the model’s outputs to human raters, who choose which of two responses is better. Train a reward model on those choices, then optimise the language model against it: this is RLHF. Direct preference optimisation reaches a similar destination without a separate reward model, and is now common because it is simpler and cheaper.

This stage sets tone, formatting, refusal behaviour, hedging, how much detail arrives unasked. Most of what makes two models feel different to use is decided here.

What preference training also does

The objective is human approval, and humans approve of agreement. So preference training reliably produces sycophancy: agreeing with a stated position, folding when a correct answer is pushed back on, praising a mediocre plan its author is clearly attached to. This is measurable, it is documented by the labs themselves, and at least one deployed model has been rolled back for it. It is not a defect in a particular model so much as a pull inherent in the objective.

A second effect: raters prefer confident answers to hedged ones, which shifts models toward stating things plainly whether or not the underlying probability supports it. Lesson nine returns to this.

The mistake this lesson exists to prevent

A team has 5,000 pages of internal documentation. They fine-tune an open model on it so staff can ask questions. The result: the output sounds exactly like their documentation, and confidently gets specifics wrong.

This is the expected outcome, not bad luck. Gradient updates over a modest corpus reliably move style, format and register, because those are consistent patterns present on every page. Individual facts are stored diffusely across billions of weights, and a short run over a small corpus mostly teaches the model to imitate the voice of the material. Worse, the fine-tune makes it sound authoritative on exactly the subject where it is now unreliable.

Use retrieval for facts: search the documents, put the relevant passages in the prompt, ask the model to answer from them and to say when they do not contain the answer. It is cheaper, it updates the moment a document changes, and it can cite.

Use fine-tuning for behaviour: a house format, a rigid output schema, a specialised classification task, a tone. Those are the things a small number of examples teaches.

The short rule: fine-tune for form, retrieve for facts.