{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"measuring-uncertainty","title":"Measuring what the model does not know","url":"https://addaly.com/learn/how-llms-work/measuring-uncertainty","position":50,"of":78,"minutes":9,"answer":"Uncertainty has to be measured at the level of meaning rather than tokens, it only becomes a probability after calibration against labelled outcomes.","keyIdea":"Uncertainty has to be measured at the level of meaning rather than tokens, it only becomes a probability after calibration against labelled outcomes, and no measure of uncertainty can catch an answer the model is confidently wrong about."},"body":{"format":"markdown","text":"## The signal exists; the problem is reading it\n\nThe previous lesson ended on a finding worth taking seriously: somewhere inside the network there is usually information that correlates with whether the model is about to be wrong. Module three's lesson on log-probabilities gave you the crudest way to read it, and its caveat: good for ranking, unreliable as an absolute number. This lesson is about the four methods that do better, what each costs, and the one failure none of them can catch.\n\n## Token probability is the wrong unit\n\nAsk a model the capital of Australia and look at the probability of the first answer token. It is low. Not because the model is unsure of the answer, but because the answer can start with \"Canberra\", \"The\", \"Australia's\" or \"It\", and the probability is spread across phrasings.\n\nThis is the mechanical reason raw logprobs mislead. They measure uncertainty about the **next token**, which mixes two things: uncertainty about *what to say* and uncertainty about *how to say it*. A long answer contains dozens of low-probability tokens that are synonym choices, and a single confidently wrong fact stated in the most probable words.\n\nEvery method below is an attempt to strip the phrasing out and leave the meaning.\n\n## Semantic entropy\n\nThe cleanest current method, published in 2024, does the obvious thing well.\n\n1. Sample the answer several times, five to ten, at a temperature around 1.\n2. Group the samples by **meaning**, not by string. Two answers go in the same group if each entails the other. A small natural-language-inference model does this, or a cheap language model asked \"do these two say the same thing?\".\n3. Compute the entropy over the groups, weighted by how much probability each group received.\n\nIf nine samples say Canberra in nine phrasings, there is one group and the entropy is zero. If four say Canberra, three say Sydney and two say Melbourne, there are three groups and the entropy is high. The second case is the model telling you it does not know, in the only language it has.\n\nOn short factual questions this separates right from wrong answers markedly better than raw logprobs, across model families. The cost is real: five to ten times the tokens per question, plus the grouping step. For long-form answers you first have to split the text into individual claims and score each, which is slower still.\n\n## Asking the model\n\nTwo cheaper methods, honestly rated.\n\n**Verbalised confidence.** Append \"how confident are you, from 0 to 100?\". The numbers cluster between 80 and 95 almost regardless of the question, because preference training rewarded sounding sure. The ranking has some information in it; the values have almost none. Treat a verbalised 60 as \"lower than usual\", never as a probability.\n\n**P(True).** Show the model its own answer and ask whether it is correct, then read the log-probability of the token \"True\" rather than the text of the reply. This is better than verbalised confidence because you are reading a distribution instead of a sampled word, and it tends to be better calibrated on base models than on chat-tuned ones. It still needs the calibration step below before the number means anything.\n\n## Earning the right to call it a probability\n\nWhichever score you choose, it becomes a probability only after you have measured it against outcomes. That takes a labelled set (two to five hundred questions with known answers is enough to start) and a few lines of code.\n\n```python\nimport numpy as np\n# conf: the model's score in [0,1]; correct: 1 or 0, from your labels\nedges = np.linspace(0, 1, 11)\nfor lo, hi in zip(edges[:-1], edges[1:]):\n    m = (conf >= lo) & (conf < hi)\n    if m.sum():\n        print(f\"{lo:.1f}-{hi:.1f}  n={m.sum():4d}  accuracy={correct[m].mean():.2f}\")\n```\n\nIf the 0.9 bucket is right 90 per cent of the time, you have a calibrated score. If it is right 60 per cent of the time (the usual result for a chat model) you rescale: fit a single temperature or an isotonic curve that maps claimed confidence onto observed accuracy. The rescaling is a small model of the model, and it drifts when the underlying model changes, so it is re-fitted every time the model is.\n\n## What a score is for\n\nAn uncertainty score is a routing decision. Below the threshold: abstain, retrieve more, escalate to a larger model, or hand to a person. The threshold comes from the calibration curve and from the cost of a wrong answer: a support bot can afford to guess where a medical intake form cannot. Choosing it is a product decision that the curve informs and does not make.\n\n## The failure no method catches\n\nAll four methods measure the model's **uncertainty**. None measures **truth**. A model trained on a widespread misconception is certain and wrong: it will produce the same false answer ten times in ten phrasings, semantic entropy will be zero, P(True) will be high, and every routing rule above will wave it through.\n\nThat is not a weakness of a particular method. It is what the methods are: readings of the distribution the model produced, and the distribution has no access to the world. Uncertainty scoring finds the thin ice. Only grounding (a retrieved source, a tool, a person) finds the confident mistakes, and a system that needs both should have both."},"check":{"question":"A system uses semantic entropy to flag unreliable answers. On a question where the model gives the same wrong answer in every sample, the answer passes the check. Why?","options":[{"text":"Semantic entropy is only defined for numerical answers, so free-text questions are not scored correctly.","correct":false,"tempting":"Invents a scope restriction; the method groups any answers by mutual entailment, and short factual text is its best-studied case."},{"text":"The entailment step merged the wrong answer with the right one into a single group.","correct":false,"tempting":"A real failure mode of the grouping step, but it would require a correct answer to be present among the samples, and here every sample is the same wrong answer."},{"text":"Low entropy means the model is sure, and a memorised falsehood makes it sure and wrong.","correct":true},{"text":"The sampling temperature was set too low, so every sample was identical and the entropy was artificially zero.","correct":false,"tempting":"Temperature does affect the spread, which makes this plausible, but the lesson's point is that at a proper temperature a confidently held belief still produces one meaning-group, because the distribution itself is peaked."}]},"prev":{"slug":"hallucination-by-design","title":"Why hallucination is a property of the method","url":"https://addaly.com/learn/how-llms-work/hallucination-by-design"},"next":{"slug":"sycophancy","title":"Sycophancy: agreement as a trained reflex","url":"https://addaly.com/learn/how-llms-work/sycophancy"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring what the model does not know', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/measuring-uncertainty (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring what the model does not know","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"Uncertainty has to be measured at the level of meaning rather than tokens, it only becomes a probability after calibration against labelled outcomes, and no measure of uncertainty can catch an answer the model is confidently wrong about.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-signal-exists-the-problem-is-reading-it","runs":[{"text":"The signal exists; the problem is reading it"}]},{"t":"p","runs":[{"text":"The previous lesson ended on a finding worth taking seriously: somewhere inside the network there is usually information that correlates with whether the model is about to be wrong. Module three’s lesson on log-probabilities gave you the crudest way to read it, and its caveat: good for ranking, unreliable as an absolute number. This lesson is about the four methods that do better, what each costs, and the one failure none of them can catch."}]},{"t":"h2","id":"token-probability-is-the-wrong-unit","runs":[{"text":"Token probability is the wrong unit"}]},{"t":"p","runs":[{"text":"Ask a model the capital of Australia and look at the probability of the first answer token. It is low. Not because the model is unsure of the answer, but because the answer can start with “Canberra”, “The”, “Australia’s” or “It”, and the probability is spread across phrasings."}]},{"t":"p","runs":[{"text":"This is the mechanical reason raw logprobs mislead. They measure uncertainty about the "},{"text":"next token","b":true},{"text":", which mixes two things: uncertainty about "},{"text":"what to say","i":true},{"text":" and uncertainty about "},{"text":"how to say it","i":true},{"text":". A long answer contains dozens of low-probability tokens that are synonym choices, and a single confidently wrong fact stated in the most probable words."}]},{"t":"p","runs":[{"text":"Every method below is an attempt to strip the phrasing out and leave the meaning."}]},{"t":"h2","id":"semantic-entropy","runs":[{"text":"Semantic entropy"}]},{"t":"p","runs":[{"text":"The cleanest current method, published in 2024, does the obvious thing well."}]},{"t":"ol","start":1,"items":[[{"text":"Sample the answer several times, five to ten, at a temperature around 1."}],[{"text":"Group the samples by "},{"text":"meaning","b":true},{"text":", not by string. Two answers go in the same group if each entails the other. A small natural-language-inference model does this, or a cheap language model asked “do these two say the same thing?”."}],[{"text":"Compute the entropy over the groups, weighted by how much probability each group received."}]]},{"t":"p","runs":[{"text":"If nine samples say Canberra in nine phrasings, there is one group and the entropy is zero. If four say Canberra, three say Sydney and two say Melbourne, there are three groups and the entropy is high. The second case is the model telling you it does not know, in the only language it has."}]},{"t":"p","runs":[{"text":"On short factual questions this separates right from wrong answers markedly better than raw logprobs, across model families. The cost is real: five to ten times the tokens per question, plus the grouping step. For long-form answers you first have to split the text into individual claims and score each, which is slower still."}]},{"t":"h2","id":"asking-the-model","runs":[{"text":"Asking the model"}]},{"t":"p","runs":[{"text":"Two cheaper methods, honestly rated."}]},{"t":"p","runs":[{"text":"Verbalised confidence.","b":true},{"text":" Append “how confident are you, from 0 to 100?”. The numbers cluster between 80 and 95 almost regardless of the question, because preference training rewarded sounding sure. The ranking has some information in it; the values have almost none. Treat a verbalised 60 as “lower than usual”, never as a probability."}]},{"t":"p","runs":[{"text":"P(True).","b":true},{"text":" Show the model its own answer and ask whether it is correct, then read the log-probability of the token “True” rather than the text of the reply. This is better than verbalised confidence because you are reading a distribution instead of a sampled word, and it tends to be better calibrated on base models than on chat-tuned ones. It still needs the calibration step below before the number means anything."}]},{"t":"h2","id":"earning-the-right-to-call-it-a-probability","runs":[{"text":"Earning the right to call it a probability"}]},{"t":"p","runs":[{"text":"Whichever score you choose, it becomes a probability only after you have measured it against outcomes. That takes a labelled set (two to five hundred questions with known answers is enough to start) and a few lines of code."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\n# conf: the model's score in [0,1]; correct: 1 or 0, from your labels\nedges = np.linspace(0, 1, 11)\nfor lo, hi in zip(edges[:-1], edges[1:]):\n    m = (conf >= lo) & (conf < hi)\n    if m.sum():\n        print(f\"{lo:.1f}-{hi:.1f}  n={m.sum():4d}  accuracy={correct[m].mean():.2f}\")"},{"t":"p","runs":[{"text":"If the 0.9 bucket is right 90 per cent of the time, you have a calibrated score. If it is right 60 per cent of the time (the usual result for a chat model) you rescale: fit a single temperature or an isotonic curve that maps claimed confidence onto observed accuracy. The rescaling is a small model of the model, and it drifts when the underlying model changes, so it is re-fitted every time the model is."}]},{"t":"h2","id":"what-a-score-is-for","runs":[{"text":"What a score is for"}]},{"t":"p","runs":[{"text":"An uncertainty score is a routing decision. Below the threshold: abstain, retrieve more, escalate to a larger model, or hand to a person. The threshold comes from the calibration curve and from the cost of a wrong answer: a support bot can afford to guess where a medical intake form cannot. Choosing it is a product decision that the curve informs and does not make."}]},{"t":"h2","id":"the-failure-no-method-catches","runs":[{"text":"The failure no method catches"}]},{"t":"p","runs":[{"text":"All four methods measure the model’s "},{"text":"uncertainty","b":true},{"text":". None measures "},{"text":"truth","b":true},{"text":". A model trained on a widespread misconception is certain and wrong: it will produce the same false answer ten times in ten phrasings, semantic entropy will be zero, P(True) will be high, and every routing rule above will wave it through."}]},{"t":"p","runs":[{"text":"That is not a weakness of a particular method. It is what the methods are: readings of the distribution the model produced, and the distribution has no access to the world. Uncertainty scoring finds the thin ice. Only grounding (a retrieved source, a tool, a person) finds the confident mistakes, and a system that needs both should have both."}]}],"check":{"question":[{"text":"A system uses semantic entropy to flag unreliable answers. On a question where the model gives the same wrong answer in every sample, the answer passes the check. Why?"}],"options":[{"text":[{"text":"Semantic entropy is only defined for numerical answers, so free-text questions are not scored correctly."}],"correct":false,"tempting":[{"text":"Invents a scope restriction; the method groups any answers by mutual entailment, and short factual text is its best-studied case."}]},{"text":[{"text":"The entailment step merged the wrong answer with the right one into a single group."}],"correct":false,"tempting":[{"text":"A real failure mode of the grouping step, but it would require a correct answer to be present among the samples, and here every sample is the same wrong answer."}]},{"text":[{"text":"Low entropy means the model is sure, and a memorised falsehood makes it sure and wrong."}],"correct":true,"tempting":null},{"text":[{"text":"The sampling temperature was set too low, so every sample was identical and the entropy was artificially zero."}],"correct":false,"tempting":[{"text":"Temperature does affect the spread, which makes this plausible, but the lesson’s point is that at a proper temperature a confidently held belief still produces one meaning-group, because the distribution itself is peaked."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"hallucination-by-design","title":"Why hallucination is a property of the method","minutes":8},"next":{"slug":"sycophancy","title":"Sycophancy: agreement as a trained reflex","minutes":8}}}