{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"reading-logprobs","title":"Reading the numbers the model will give you","url":"https://addaly.com/learn/how-llms-work/reading-logprobs","position":23,"of":78,"minutes":9,"answer":"Log-probabilities are the only quantitative signal a model exposes about its own output, and they are useful for ranking even though they are unreliable as absolute confidence.","keyIdea":"Log-probabilities are the only quantitative signal a model exposes about its own output, and they are useful for ranking even though they are unreliable as absolute confidence."},"body":{"format":"markdown","text":"## The one honest number\n\nAsk a model how confident it is and it produces text: the word \"95%\" generated the same way as any other word, with no connection to anything internal. But the model does expose a real number: the probability it assigned to each token it chose, and to the alternatives it did not.\n\nMost APIs return this on request, and every local library gives it to you for free.\n\n```python\nresp = client.chat.completions.create(\n    model=\"...\", messages=msgs, logprobs=True, top_logprobs=5,\n)\nfor t in resp.choices[0].logprobs.content:\n    print(t.token, round(t.logprob, 2),\n          [(a.token, round(a.logprob, 2)) for a in t.top_logprobs])\n```\n\nLog-probabilities are negative. Zero means certainty; −0.02 is about 98 per cent; −2.3 is about 10 per cent. Exponentiate to get back to a probability.\n\n## Four things you can actually do with them\n\n**1. Classification with a real score.** Constrain the model to answer with one token (`yes` or `no`, or a single category label) and read the probabilities of the candidates at that position. Now you have a graded score instead of a word, so you can set a threshold, rank items for review, and route only the uncertain ones to a human. This is the single most useful application, and it converts a chat model into something you can tune.\n\n**2. Finding where a generation went wrong.** Print the per-token log-probabilities of a long answer and look for the low points. A confidently generated sentence containing an invented citation will often show a sharp dip exactly at the fabricated name: the model had many plausible options and low probability on each. It is not a reliable detector on its own, but as a triage signal for review it earns its cost.\n\n**3. Scoring candidates you already have.** You can ask for the log-probability of a *given* text rather than generating one, which lets you rank several rewrites, pick between two translations, or measure whether one phrasing is more natural under the model than another.\n\n**4. Perplexity.** The exponential of the average negative log-probability. It is the standard measure of how surprised a model is by text, and you will meet it again in module five as the training objective's direct readout.\n\n## The calibration problem, stated honestly\n\nNow the caveat, which is important enough that ignoring it produces bad systems.\n\nA **base** model (pretrained only) tends to be reasonably well calibrated. When it says 70 per cent, the answer is right about 70 per cent of the time, because that is precisely what its training objective rewarded.\n\nPreference training damages this. The published evidence is clear on the direction: models tuned to be helpful and agreeable become **overconfident**, pushing probability mass toward whichever answer reads as decisive. Some of the calibration can be recovered by rescaling, and some cannot.\n\nSo: log-probabilities are good for **ranking** and unreliable as **absolute** confidence. \"Item A scored higher than item B\" is usable. \"The model is 91 per cent sure\" is not, unless you have measured the calibration on your own data, which means bucketing your predictions by claimed confidence and checking the actual accuracy in each bucket. That is an afternoon of work and it is the only way to earn the right to use the number as a probability.\n\n## Two practical traps\n\n**Tokenisation interferes.** If your category labels tokenise into different numbers of tokens (`yes` is one token, `maybe not` is three) the comparison is unfair, because you are comparing a single probability against a product of three. Choose single-token labels, or normalise by length.\n\n**Not every provider exposes them.** Some reasoning-oriented endpoints withhold log-probabilities entirely. If a confidence signal is load-bearing in your design, confirm availability before you build on it, and keep an open-weights fallback where you always have access.\n\n## Why this belongs in a mechanics course\n\nEverything else about confidence is inference from behaviour. This is the one place where you can read a number that the model actually computed. Knowing it exists, knowing what it is good for, and knowing exactly where it stops being trustworthy is the difference between a system that routes uncertain cases sensibly and one that trusts a sentence saying \"I am confident\"."},"check":{"question":"A team uses a model's stated \"I am 95% confident\" text as a threshold for auto-approving claims, and finds the accuracy at that level is nearer 70 per cent. What would fix the signal?","options":[{"text":"Instruct the model to be more conservative in its stated confidence.","correct":false,"tempting":"Prompting for calibration is the obvious first move and does shift the numbers, without making them correspond to anything measured."},{"text":"Use a larger model, since bigger models are better calibrated.","correct":false,"tempting":"Scale does improve many things, and base-model calibration does improve with scale, which makes this a defensible-sounding generalisation."},{"text":"Constrain the answer to a single token and read its log-probability, then check calibration by bucketing past predictions and measuring accuracy in each bucket.","correct":true},{"text":"Average the stated confidence across five samples of the same prompt.","correct":false,"tempting":"Averaging across samples is a genuinely useful variance-reduction trick, so applying it here feels methodologically sound."}]},"prev":{"slug":"temperature-and-sampling","title":"Temperature and sampling: what the knob really does","url":"https://addaly.com/learn/how-llms-work/temperature-and-sampling"},"next":{"slug":"stop-tokens-and-chat-templates","title":"Special tokens, stop conditions, and where a turn ends","url":"https://addaly.com/learn/how-llms-work/stop-tokens-and-chat-templates"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reading the numbers the model will give you', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/reading-logprobs (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reading the numbers the model will give you","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Log-probabilities are the only quantitative signal a model exposes about its own output, and they are useful for ranking even though they are unreliable as absolute confidence.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-one-honest-number","runs":[{"text":"The one honest number"}]},{"t":"p","runs":[{"text":"Ask a model how confident it is and it produces text: the word “95%” generated the same way as any other word, with no connection to anything internal. But the model does expose a real number: the probability it assigned to each token it chose, and to the alternatives it did not."}]},{"t":"p","runs":[{"text":"Most APIs return this on request, and every local library gives it to you for free."}]},{"t":"code","lang":"python","label":"Python","text":"resp = client.chat.completions.create(\n    model=\"...\", messages=msgs, logprobs=True, top_logprobs=5,\n)\nfor t in resp.choices[0].logprobs.content:\n    print(t.token, round(t.logprob, 2),\n          [(a.token, round(a.logprob, 2)) for a in t.top_logprobs])"},{"t":"p","runs":[{"text":"Log-probabilities are negative. Zero means certainty; −0.02 is about 98 per cent; −2.3 is about 10 per cent. Exponentiate to get back to a probability."}]},{"t":"h2","id":"four-things-you-can-actually-do-with-them","runs":[{"text":"Four things you can actually do with them"}]},{"t":"p","runs":[{"text":"1. Classification with a real score.","b":true},{"text":" Constrain the model to answer with one token ("},{"text":"yes","code":true},{"text":" or "},{"text":"no","code":true},{"text":", or a single category label) and read the probabilities of the candidates at that position. Now you have a graded score instead of a word, so you can set a threshold, rank items for review, and route only the uncertain ones to a human. This is the single most useful application, and it converts a chat model into something you can tune."}]},{"t":"p","runs":[{"text":"2. Finding where a generation went wrong.","b":true},{"text":" Print the per-token log-probabilities of a long answer and look for the low points. A confidently generated sentence containing an invented citation will often show a sharp dip exactly at the fabricated name: the model had many plausible options and low probability on each. It is not a reliable detector on its own, but as a triage signal for review it earns its cost."}]},{"t":"p","runs":[{"text":"3. Scoring candidates you already have.","b":true},{"text":" You can ask for the log-probability of a "},{"text":"given","i":true},{"text":" text rather than generating one, which lets you rank several rewrites, pick between two translations, or measure whether one phrasing is more natural under the model than another."}]},{"t":"p","runs":[{"text":"4. Perplexity.","b":true},{"text":" The exponential of the average negative log-probability. It is the standard measure of how surprised a model is by text, and you will meet it again in module five as the training objective’s direct readout."}]},{"t":"h2","id":"the-calibration-problem-stated-honestly","runs":[{"text":"The calibration problem, stated honestly"}]},{"t":"p","runs":[{"text":"Now the caveat, which is important enough that ignoring it produces bad systems."}]},{"t":"p","runs":[{"text":"A "},{"text":"base","b":true},{"text":" model (pretrained only) tends to be reasonably well calibrated. When it says 70 per cent, the answer is right about 70 per cent of the time, because that is precisely what its training objective rewarded."}]},{"t":"p","runs":[{"text":"Preference training damages this. The published evidence is clear on the direction: models tuned to be helpful and agreeable become "},{"text":"overconfident","b":true},{"text":", pushing probability mass toward whichever answer reads as decisive. Some of the calibration can be recovered by rescaling, and some cannot."}]},{"t":"p","runs":[{"text":"So: log-probabilities are good for "},{"text":"ranking","b":true},{"text":" and unreliable as "},{"text":"absolute","b":true},{"text":" confidence. “Item A scored higher than item B” is usable. “The model is 91 per cent sure” is not, unless you have measured the calibration on your own data, which means bucketing your predictions by claimed confidence and checking the actual accuracy in each bucket. That is an afternoon of work and it is the only way to earn the right to use the number as a probability."}]},{"t":"h2","id":"two-practical-traps","runs":[{"text":"Two practical traps"}]},{"t":"p","runs":[{"text":"Tokenisation interferes.","b":true},{"text":" If your category labels tokenise into different numbers of tokens ("},{"text":"yes","code":true},{"text":" is one token, "},{"text":"maybe not","code":true},{"text":" is three) the comparison is unfair, because you are comparing a single probability against a product of three. Choose single-token labels, or normalise by length."}]},{"t":"p","runs":[{"text":"Not every provider exposes them.","b":true},{"text":" Some reasoning-oriented endpoints withhold log-probabilities entirely. If a confidence signal is load-bearing in your design, confirm availability before you build on it, and keep an open-weights fallback where you always have access."}]},{"t":"h2","id":"why-this-belongs-in-a-mechanics-course","runs":[{"text":"Why this belongs in a mechanics course"}]},{"t":"p","runs":[{"text":"Everything else about confidence is inference from behaviour. This is the one place where you can read a number that the model actually computed. Knowing it exists, knowing what it is good for, and knowing exactly where it stops being trustworthy is the difference between a system that routes uncertain cases sensibly and one that trusts a sentence saying “I am confident”."}]}],"check":{"question":[{"text":"A team uses a model’s stated “I am 95% confident” text as a threshold for auto-approving claims, and finds the accuracy at that level is nearer 70 per cent. What would fix the signal?"}],"options":[{"text":[{"text":"Instruct the model to be more conservative in its stated confidence."}],"correct":false,"tempting":[{"text":"Prompting for calibration is the obvious first move and does shift the numbers, without making them correspond to anything measured."}]},{"text":[{"text":"Use a larger model, since bigger models are better calibrated."}],"correct":false,"tempting":[{"text":"Scale does improve many things, and base-model calibration does improve with scale, which makes this a defensible-sounding generalisation."}]},{"text":[{"text":"Constrain the answer to a single token and read its log-probability, then check calibration by bucketing past predictions and measuring accuracy in each bucket."}],"correct":true,"tempting":null},{"text":[{"text":"Average the stated confidence across five samples of the same prompt."}],"correct":false,"tempting":[{"text":"Averaging across samples is a genuinely useful variance-reduction trick, so applying it here feels methodologically sound."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"temperature-and-sampling","title":"Temperature and sampling: what the knob really does","minutes":6},"next":{"slug":"stop-tokens-and-chat-templates","title":"Special tokens, stop conditions, and where a turn ends","minutes":8}}}