Disagreeing without getting agreement

The check that destroys the thing it checks

You suspect an answer is wrong. You ask: are you sure? The model apologises and produces a different answer.

You have learned nothing. Preference training rewarded responses people approved of, and disagreeing with the person asking is not what raters approved of. The apology and the revision are the trained behaviour, not a reconsideration.

The uncomfortable part: this works in both directions. Push back on a correct answer and you can get it replaced with a wrong one. People have talked models out of the right answer countless times without noticing, because the reversal feels like the model conceding to a good argument.

Questions that leak the answer you want

Each of these tells the model what you would like to hear:

• “Are you sure? I thought it was 1974.”
• “That doesn’t sound right.”
• “Isn’t it actually the other way round?”
• “My colleague says X.”
• “Can you double-check? I need this to be right.”

Even the last one, which feels neutral, signals that the current answer is under suspicion.

Questions that do not

Ask for the case against.

What is the strongest argument that this answer is wrong?

Ask for the evidence, not the conclusion.

What specifically supports this? Quote it.

Offer alternatives symmetrically, without a preference.

Two candidate answers: 1974 and 1979. Give the evidence for each, then say which is better supported and why.

Ask what would change it.

What would I need to find to show this is wrong?

Start again in a clean context. Paste the same question in a fresh conversation with no history of your doubt. If the answer comes back the same, that is weak independent evidence. If it comes back different, you have found instability worth investigating.

The related trap: agreement with your framing

Sycophancy is not only about facts. It also shows up as accepting your premises.

Ask “why is our churn so high?” and you get reasons churn might be high, whether or not it is. The question smuggled in a claim, and the shape of the exchange is to answer rather than to challenge.

The fix is to ask the prior question separately:

Here is the churn data. Is it high, compared with what? Then, only if it is, what might explain it.

This is worth a habit, because in professional work the smuggled premise is more common and more expensive than the wrong fact. Every “how do we fix X” contains a claim that X is broken.

Where models push back

To be fair to them: current models will contradict you on matters of fact more often than earlier ones, and will decline to agree that 2+2=5. Providers have measured sycophancy, published on it, and worked to reduce it. It is better than it was.

It has not gone away, and it is strongest exactly where you are least able to detect it: judgement calls, plans, drafts, and anything where you have visibly invested effort. A model reviewing your work is a system trained towards approval, reading something you clearly want approved.

The practical rule

Never verify by asking whether it is sure. Verify by asking for the evidence, by offering alternatives without a preference, by asking for the case against, or by asking again in a clean context.

And when a model does agree with you about something important, notice that agreement is the cheapest thing it produces. It is not confirmation. It is the default.

A test you can run on yourself

Take a factual question you know the answer to: a date, a rule in your field, a figure from your own work. Ask it cold and get the correct answer. Then push back: are you sure? I thought it was different.

Watch what happens. Some models hold; many will hedge, apologise, or offer your version as equally plausible.

Do this once and the lesson stops being abstract. It also inoculates you against the moment that matters, which is when you push back on something you are wrong about and receive immediate, articulate agreement; and have no way of telling that apart from being right.