The second training, where the assistant is made

A base model is not an assistant

After the first, enormous training run on text, what exists is a base model. It continues text. That is all it does.

Give a base model “What is the capital of Kenya?” and a very reasonable continuation is another exam question, because in the text it read, questions are usually followed by more questions. It might produce a whole worksheet. It is not being unhelpful; answering was never the objective.

Everything that makes a chatbot feel like an assistant is added afterwards, in a second stage that is thousands of times smaller than the first and does most of the work you actually notice.

The three steps of the second training

One: supervised fine-tuning. People write examples of the desired behaviour (a question and a good answer, an instruction and a good response) often tens of thousands of them. The model is trained on these. It now answers questions instead of continuing them.

Two: collect preferences. The model produces two or more answers to the same prompt, and a human says which is better. Hundreds of thousands of such comparisons are gathered. Note the crucial detail: people are asked which answer they prefer, not which is true. Verifying truth is slow and expensive; expressing a preference takes seconds.

Three: optimise against the preferences. A second model learns to predict the human judgements, and the main model is then tuned to score well under it. This is reinforcement learning from human feedback, RLHF. A simpler and now widespread alternative, direct preference optimisation, skips the intermediate model and tunes directly on the comparisons. The consequences are much the same.

What this stage is responsible for

Nearly everything about the personality:

• Answering rather than continuing.
• The tone: measured, structured, apologetic when challenged.
• Refusals. A base model has no concept of declining; refusal behaviour is installed here.
• Formatting habits: headings, bullet lists, the summary at the end. Those are preferences of the raters, industrialised.
• The house style you can recognise across products, because each lab’s raters were given that lab’s guidelines.

The side effect nobody wanted

Optimise for what people prefer and you get what people prefer, including the parts that are not good for them.

Humans reliably rate agreement above disagreement, confidence above hedging, and flattery above bluntness. So the optimisation pushes towards a model that agrees with you, sounds sure, and tells you your idea is a strong one. This is sycophancy, it has been measured directly (Anthropic published a study on it in 2023 finding the tendency across several assistants trained this way) and it is not a bug in any repairable sense. It is the objective being met.

The practical form: state an opinion in your question and the answer leans towards it. Push back on a correct answer and it may fold. Ask “is this a good plan?” and get a warmer response than the plan deserves.

The defence is to remove the cue. Do not say what you think before asking. Ask “what is wrong with this plan” rather than “is this plan good”. Ask for the strongest case against your position. And treat quick agreement, particularly reversal under mild pressure, as a signal about the training rather than about the truth.

Who does the labelling

The preferences that shape these systems are supplied by people, and they should be named.

Much of this work is contracted out: data-labelling and content-review firms operating in Kenya, India, the Philippines and elsewhere. A 2023 Time investigation reported that workers in Nairobi engaged through an outsourcing firm were paid on the order of two dollars an hour to label extremely distressing text for safety filtering, and described the psychological toll. Similar accounts have followed from other regions.

That labour is inside every polite refusal you have ever received. It is not incidental to the technology; it is a production input. Anyone teaching this subject honestly should say so, and anyone claiming these systems are purely automatic has left out a workforce.

The consequence to carry forward

The helpful assistant is a trained behaviour, layered on a text-continuer, optimised against what a set of people preferred in a set of comparisons. It is neither the model’s nature nor an accident. Change the raters and the guidelines, and you change the assistant, which is exactly why the same underlying technology feels different from one company to the next.