Measuring what the model does not know

The signal exists; the problem is reading it

The previous lesson ended on a finding worth taking seriously: somewhere inside the network there is usually information that correlates with whether the model is about to be wrong. Module three’s lesson on log-probabilities gave you the crudest way to read it, and its caveat: good for ranking, unreliable as an absolute number. This lesson is about the four methods that do better, what each costs, and the one failure none of them can catch.

Token probability is the wrong unit

Ask a model the capital of Australia and look at the probability of the first answer token. It is low. Not because the model is unsure of the answer, but because the answer can start with “Canberra”, “The”, “Australia’s” or “It”, and the probability is spread across phrasings.

This is the mechanical reason raw logprobs mislead. They measure uncertainty about the next token, which mixes two things: uncertainty about what to say and uncertainty about how to say it. A long answer contains dozens of low-probability tokens that are synonym choices, and a single confidently wrong fact stated in the most probable words.

Every method below is an attempt to strip the phrasing out and leave the meaning.

Semantic entropy

The cleanest current method, published in 2024, does the obvious thing well.

1. Sample the answer several times, five to ten, at a temperature around 1.
2. Group the samples by meaning, not by string. Two answers go in the same group if each entails the other. A small natural-language-inference model does this, or a cheap language model asked “do these two say the same thing?”.
3. Compute the entropy over the groups, weighted by how much probability each group received.

If nine samples say Canberra in nine phrasings, there is one group and the entropy is zero. If four say Canberra, three say Sydney and two say Melbourne, there are three groups and the entropy is high. The second case is the model telling you it does not know, in the only language it has.

On short factual questions this separates right from wrong answers markedly better than raw logprobs, across model families. The cost is real: five to ten times the tokens per question, plus the grouping step. For long-form answers you first have to split the text into individual claims and score each, which is slower still.

Asking the model

Two cheaper methods, honestly rated.

Verbalised confidence. Append “how confident are you, from 0 to 100?”. The numbers cluster between 80 and 95 almost regardless of the question, because preference training rewarded sounding sure. The ranking has some information in it; the values have almost none. Treat a verbalised 60 as “lower than usual”, never as a probability.

P(True). Show the model its own answer and ask whether it is correct, then read the log-probability of the token “True” rather than the text of the reply. This is better than verbalised confidence because you are reading a distribution instead of a sampled word, and it tends to be better calibrated on base models than on chat-tuned ones. It still needs the calibration step below before the number means anything.

Earning the right to call it a probability

Whichever score you choose, it becomes a probability only after you have measured it against outcomes. That takes a labelled set (two to five hundred questions with known answers is enough to start) and a few lines of code.

import numpy as np
# conf: the model's score in [0,1]; correct: 1 or 0, from your labels
edges = np.linspace(0, 1, 11)
for lo, hi in zip(edges[:-1], edges[1:]):
    m = (conf >= lo) & (conf < hi)
    if m.sum():
        print(f"{lo:.1f}-{hi:.1f}  n={m.sum():4d}  accuracy={correct[m].mean():.2f}")

If the 0.9 bucket is right 90 per cent of the time, you have a calibrated score. If it is right 60 per cent of the time (the usual result for a chat model) you rescale: fit a single temperature or an isotonic curve that maps claimed confidence onto observed accuracy. The rescaling is a small model of the model, and it drifts when the underlying model changes, so it is re-fitted every time the model is.

What a score is for

An uncertainty score is a routing decision. Below the threshold: abstain, retrieve more, escalate to a larger model, or hand to a person. The threshold comes from the calibration curve and from the cost of a wrong answer: a support bot can afford to guess where a medical intake form cannot. Choosing it is a product decision that the curve informs and does not make.

The failure no method catches

All four methods measure the model’s uncertainty. None measures truth. A model trained on a widespread misconception is certain and wrong: it will produce the same false answer ten times in ten phrasings, semantic entropy will be zero, P(True) will be high, and every routing rule above will wave it through.

That is not a weakness of a particular method. It is what the methods are: readings of the distribution the model produced, and the distribution has no access to the world. Uncertainty scoring finds the thin ice. Only grounding (a retrieved source, a tool, a person) finds the confident mistakes, and a system that needs both should have both.