Why it obeys you at all

A raw model does not answer questions

Take a model straight out of pretraining (the stage where it learned to continue text scraped from the web and from books) and give it this:

Write a short poem about rain.

A raw base model may well continue:

Write a short poem about fog. Write a short poem about the sea. Submissions close on 14 March.

It has done nothing wrong. On the open internet, a line like that is usually followed by more lines like it. Base models continue documents; they do not take instructions, because taking instructions was never the objective.

The models you use have been through two more stages. First instruction tuning: training on many thousands of examples of the form here is a request, here is a good response to it. Then preference training, usually some variant of learning from human comparisons, where people were shown pairs of responses and asked which was better, and the model was nudged towards the winners.

Obedience is a trained behaviour. It is not a property of neural networks and it is not built into the architecture. That single fact explains most of the odd behaviour in this course.

What follows from it

Politeness is close to free, and close to useless. “Please” and “thank you” cost a token or two and have no reliable effect on quality. Studies that test politeness find small and inconsistent differences that flip direction between models and tasks. Say please if you like the habit. Do not budget for it.

The model is trained to be agreeable, and that has teeth. The people doing the rating preferred responses that were helpful, warm and confident. So the model leans that way, including when it should not. Push back on a correct answer (are you sure? I think it’s 1974) and a model will often fold and agree with you. This is sycophancy, and it is not a bug someone forgot to fix; it is a direct consequence of optimising for what raters liked. Providers have measured it, published on it, and reduced it, but it has not gone away.

The practical fix is to never ask “are you sure?” as a way of checking. It is a leading question and you will get agreement. Ask instead: what is the strongest argument that this answer is wrong? Or give it the alternative without saying which you prefer: someone else says 1974. Which is right and what is the evidence for each?

It will follow a bad instruction as readily as a good one. If you say “make the summary two sentences” and the document needs six, you get two sentences and lose four sentences of content. The training rewarded compliance, not the courage to tell you your instruction was wrong. Some models now push back more; none can be relied on to.

The refusals are trained too. When a model declines, that is the preference training, and the boundary is fuzzy and inconsistent by nature. Rephrasing a legitimate request often gets it through, not because you have outsmarted anything, but because you have moved the text away from the pattern the training marked as risky. A nurse asking about drug interactions may be refused once and answered when she says she is a nurse asking about drug interactions.

The part people get wrong

Because obedience is trained, it is also shallow. The model has learned the shape of following instructions, not a mechanism that guarantees it. Give it eleven constraints and it will typically satisfy eight and silently drop three, with no note that anything was dropped. Give it a constraint that contradicts another and it will pick one and continue smoothly.

There is no internal checklist. Nothing verifies the output against your request unless you build that step yourself: by asking for the check explicitly, or by checking it.

What to do with this

• Do not spend prompt space on manners, flattery or urgency. Spend it on the task.
• Never verify by asking whether the model is sure. Verify by asking what would make it wrong.
• Keep hard constraints few and checkable, and check them.
• When a model refuses something legitimate, state the real context plainly rather than trying to trick it. That is both more effective and the version you can repeat in front of a colleague.

The model is not a colleague who wants to help you and it is not an oracle that must be tricked. It is a system trained to produce the kind of text people approved of. Prompt it as what it is.