Probability is a degree of belief

Two ways to read the same number

School taught you the first one. Probability is a long-run frequency: roll a die 6000 times, expect roughly 1000 threes, so P(three) is 1/6. Something repeatable, counted.

That reading cannot handle most of what you care about. What is the probability that the rupee weakens against the dollar this quarter? That this patient has dengue rather than chikungunya? That the next word after "I went to the" is "market"? None of these repeat. There is one quarter, one patient, one sentence.

The second reading covers them. A probability is a stated degree of belief, given what you know. 0 means ruled out, 1 means certain, and everything real lives between. The number is not a property of the world. It is a property of your information about the world.

This is the reading a model uses, and once you have it, model outputs stop being mysterious.

What a model is actually saying

A language model does not output words. It outputs a belief spread across every token in its vocabulary, conditioned on everything it has seen so far.

P(next = "market"  | "I went to the") = 0.21
P(next = "office"  | "I went to the") = 0.14
P(next = "hospital"| "I went to the") = 0.06
... 50,000 more, all adding to 1.00

Two rules constrain that list. Every number is at least zero, and they add to exactly one across the full set of options. Those two rules are why softmax exists: raw model scores can be negative or huge, and softmax exponentiates them (making everything positive) then divides by the total (making them sum to one). It converts scores into a statement of belief.

The vertical bar means "given". Everything a model reports is conditional. Change the context and the belief changes, which is why the same question with a different preamble gets a different answer. That is not the model being unstable. That is the model correctly having different beliefs given different information.

Calibration is the actual test

If a probability is a belief, how do you check it?

Not on one case. If a doctor in Manila says 30% chance of dengue and it turns out to be dengue, she was not wrong. 30% things happen 30% of the time. That is what the number claims.

You check it in bulk. Collect every case where the doctor said 30%, and see whether about 3 in 10 turned out that way. Then do the same for her 70% cases and her 90% cases. If each bucket lands near its label, she is calibrated, and her numbers are worth something. If everything she calls 90% happens half the time, she is overconfident, and you should discount her.

That is the only honest test, and it applies to models exactly as written. Take 10,000 predictions the model gave 0.8 to, and count how many came true. Near 8,000 is calibrated. Near 5,000 means the model's confidence is decoration.

The uncomfortable part

Base models tend to be reasonably calibrated. Instruction-tuned and RLHF-tuned models tend to be worse — the tuning rewards answers that sound helpful and assured, and hedging reads as unhelpful, so confidence gets inflated without accuracy following it up. A model that says "I am 95% sure" in prose is producing text that scored well with raters, which is a different thing from a belief you can bet on. The token probabilities underneath are more trustworthy than the sentence, though not by as much as you would like.

So: a confident model is not a correct model. When someone shows you a system that assigns probabilities, the question is never whether the last prediction was right. It is whether anyone has plotted the buckets.