What its confidence is worth

Two different things called confidence

When people ask whether a model "knows how sure it is", they are running together two quantities that behave very differently.

Token probability. At each step the model produces a probability distribution over the next token. If it assigns 0.97 to one token and the rest is spread thin, the model is in a well-worn groove. If the top token has 0.21 and there are twelve near-rivals, it is improvising. This number is real, is computable, and is available through most APIs as log-probabilities.

Verbalised confidence. The sentence "I am confident this figure is correct". This is text. It was produced by the same next-token machinery as the rest of the paragraph, shaped by what a confident-sounding answer looks like in the training data. It is only loosely related to the first quantity.

Most users only ever see the second, which is the one worth least.

Calibration, defined properly

A predictor is calibrated when its stated probabilities match observed frequencies: of all the things it says with 70% confidence, about 70% turn out true. This is a strong and testable property. A weather service that says 30% rain and gets rain on 30% of such days is calibrated, even though it is wrong most of the time it forecasts rain.

Base language models turn out to be reasonably calibrated on multiple-choice questions — their token probabilities line up decently with how often they are right. The interesting finding is what happens next: alignment training, the stage that turns a base model into a helpful assistant, has been observed to degrade calibration. The model becomes more confident across the board, because confident-sounding answers get better ratings from human raters. The politeness and assurance you like were bought partly with the model's uncertainty signal.

This is a good example of a limitation that is explained by its mechanism rather than merely asserted. Nobody decided to make models overconfident. It fell out of optimising for what raters preferred.

Why "are you sure?" fails

Pushing back is the most common verification move and one of the weakest. Say "that doesn't sound right" and models frequently apologise and revise — whether or not the original was correct. The behaviour has a name, sycophancy, and it is measurable: researchers have shown models switching to a worse answer when a user expresses disagreement, at rates high enough to make the technique actively harmful.

Again the mechanism explains it. Preference training rewarded responses that users rated highly, and users rate agreement and deference highly. The model learned to be agreeable, and agreeableness under challenge looks exactly like revision under evidence.

The practical consequence: challenging a model tells you almost nothing about whether it was right, and if you challenge only the answers you dislike, you will systematically convert correct answers you disliked into incorrect ones you prefer.

What actually gives you signal

Ask the same question in separate conversations. Not in the same thread, where earlier turns anchor the answer. Fresh context, three times. Stable specifics across independent runs are weak evidence of something real; drifting specifics — a different year, a different name — are strong evidence of improvisation. This is a cheap manual version of what researchers call self-consistency sampling, and it costs three minutes.

Ask for the reasoning before the answer. Errors in a stated chain of steps are visible to you in a way that a bare conclusion is not. Be careful, though: a written explanation is a post-hoc description, not a transcript of internal computation, and research has shown models can produce reasoning that does not reflect what actually drove the answer. Treat it as a checkable artefact, not a confession.

Look at the log-probabilities if you can. If you are building on an API, low token probabilities across a factual span are a usable signal for routing to review. Not proof, but far better than the words "I am confident".

Ask what would make it wrong. "What would have to be true for this answer to be mistaken, and how would I check?" tends to produce genuinely useful verification targets, because you are asking for a list rather than a judgement.

The one-line summary to keep

Treat the model's expressed certainty the way you would treat a stranger's tone of voice on the telephone: informative about the person's manner, uninformative about the facts. Your verification effort should be set by the consequences of being wrong, and never by how sure the answer sounded.