{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"what-the-machine-is-doing","title":"What the model computes","url":"https://addaly.com/learn/ai-and-you/what-the-machine-is-doing","position":2,"of":73,"minutes":4,"answer":"Next-token prediction can produce useful patterns and plausible errors. Trace factual claims to an independent source.","keyIdea":"Next-token prediction can produce useful patterns and plausible errors. Trace factual claims to an independent source."},"body":{"format":"markdown","text":"## The answer sounds certain. Where is the evidence?\n\nOn 6 February 2023, [Google announced its experimental Bard service](https://blog.google/innovation-and-ai/technology/ai/bard-google-ai-search-updates/) and opened it to trusted testers. An early example asked for discoveries from the James Webb Space Telescope to share with a nine-year-old. The third bullet in the answer attributed the first pictures of a planet outside our solar system to Webb. The [European Southern Observatory](https://www.eso.org/public/news/eso0515/) reports that the earlier direct image of 2M1207 b was obtained in 2004 with the Very Large Telescope. Follow-up observations supported the identification in 2005. In 2022, [NASA reported Webb's own first direct image](https://science.nasa.gov/blogs/webb/2022/09/01/nasas-webb-takes-its-first-ever-direct-image-of-distant-world/) of an exoplanet. The word “own” changes what the claim means.\n\nThe answer is an example of a wrong factual claim. The example does not tell us which part of Bard's system produced it.\n\n> Figure: The missing qualifier in Bard’s answer. The February 2023 demonstration: The question asked about Webb discoveries to share with a nine-year-old.; The third bullet credited Webb with the first pictures of a planet outside our solar system.. The observatories’ record: 2004: the Very Large Telescope obtains the earlier image of 2M1207 b.; 2005: follow-up observations support the identification.; 2022: NASA reports Webb’s own first direct image of an exoplanet.. The first direct image and Webb’s own first direct image are different claims. Check the telescope, object and date before repeating either one.\n\n## Next pieces, not a list of verified facts\n\nMany text-generating language models are first trained to predict a next token from previous tokens. Tokens are pieces of text, which may be whole words, parts of words or punctuation. The model computes scores over possible next tokens. A decoding rule chooses one and the process continues. Some systems add other training objectives or steps; this is a useful sketch of a common text generation loop.\n\nPrediction can learn useful patterns about grammar, language and the world. It does not by itself look up an original source or prove that a statement is true. A deployed assistant may also retrieve documents, use tools or run checks. The reliability of those checks depends on whether they ran, what evidence they found and how the assistant used it.\n\n> Figure: A common next-token generation loop. Steps: 1. Text context (The input and text already generated.); 2. Represent text as tokens (Pieces may be words, parts of words or punctuation.); 3. Score possible continuations (The model uses its learned parameters.); 4. Choose the next token (A decoding rule selects from the scores.); 5. Continue or stop (The system can stop at an end signal or at a length limit.). This sketches text generation itself. A deployed assistant may also retrieve sources or use tools. Check whether that happened and whether the evidence supports the answer.\n\n## Where the assistant's behaviour comes from\n\n**Pretraining** teaches patterns from large amounts of data. The exact training mix and objectives depend on the model.\n\n**Instruction tuning** can teach how to respond to requests by learning from examples. **Preference training** can steer the assistant towards responses people prefer; [the InstructGPT paper](https://arxiv.org/abs/2203.02155) describes one method. In a separate set of experiments, [Sharma and colleagues](https://arxiv.org/abs/2310.13548) found that some assistants sometimes agreed with a user's stated view even when it was wrong. That does not mean every assistant will do so in every setting.\n\nModels can also show signals related to their uncertainty. [Kadavath and colleagues](https://arxiv.org/abs/2207.05221) tested ways to elicit such signals. A confident style in ordinary text is not one of those verified measurements. Treat confidence as something you test under known conditions.\n\n## Try it: open the source before using the fact\n\nWrite down one factual sentence from a generated explanation. Underline its date, named object and action. Find an independent source that could verify those details. Mark each item “supported,” “contradicted” or “not addressed.” If the source leaves a gap, keep the gap visible. A citation at the end of a paragraph is useful only when it supports the particular claim you are using."},"check":{"question":"The same assistant repeats a disputed claim and gives a confident reason. What would add independent evidence?","options":[{"text":"Ask the same question twice more.","correct":false,"tempting":"Repeated answers can repeat the same error."},{"text":"Open a source that can verify the precise claim.","correct":true},{"text":"Accept the claim because it is well phrased.","correct":false,"tempting":"Fluency and evidence are different things."},{"text":"Take agreement from a second chatbot as proof.","correct":false,"tempting":"Two models can inherit the same mistake."}]},"prev":{"slug":"what-actually-goes-wrong","title":"Design, capability or misuse","url":"https://addaly.com/learn/ai-and-you/what-actually-goes-wrong"},"next":{"slug":"training-data-is-a-choice","title":"Where the data came from","url":"https://addaly.com/learn/ai-and-you/training-data-is-a-choice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What the model computes', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/what-the-machine-is-doing (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What the model computes","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":1,"slug":"how-things-fail","title":"How these systems fail"},"answer":null,"keyIdea":"Next-token prediction can produce useful patterns and plausible errors. Trace factual claims to an independent source.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"compare","title":"The missing qualifier in Bard’s answer","caption":"The first direct image and Webb’s own first direct image are different claims. Check the telescope, object and date before repeating either one.","left":{"title":"The February 2023 demonstration","items":["The question asked about Webb discoveries to share with a nine-year-old.","The third bullet credited Webb with the first pictures of a planet outside our solar system."]},"right":{"title":"The observatories’ record","items":["2004: the Very Large Telescope obtains the earlier image of 2M1207 b.","2005: follow-up observations support the identification.","2022: NASA reports Webb’s own first direct image of an exoplanet."]}},"alt":"Figure: The missing qualifier in Bard’s answer. The February 2023 demonstration: The question asked about Webb discoveries to share with a nine-year-old.; The third bullet credited Webb with the first pictures of a planet outside our solar system.. The observatories’ record: 2004: the Very Large Telescope obtains the earlier image of 2M1207 b.; 2005: follow-up observations support the identification.; 2022: NASA reports Webb’s own first direct image of an exoplanet.. The first direct image and Webb’s own first direct image are different claims. Check the telescope, object and date before repeating either one."}},"blocks":[{"t":"h2","id":"the-answer-sounds-certain-where-is-the-evidence","runs":[{"text":"The answer sounds certain. Where is the evidence?"}]},{"t":"p","runs":[{"text":"On 6 February 2023, "},{"text":"Google announced its experimental Bard service","href":"https://blog.google/innovation-and-ai/technology/ai/bard-google-ai-search-updates/"},{"text":" and opened it to trusted testers. An early example asked for discoveries from the James Webb Space Telescope to share with a nine-year-old. The third bullet in the answer attributed the first pictures of a planet outside our solar system to Webb. The "},{"text":"European Southern Observatory","href":"https://www.eso.org/public/news/eso0515/"},{"text":" reports that the earlier direct image of 2M1207 b was obtained in 2004 with the Very Large Telescope. Follow-up observations supported the identification in 2005. In 2022, "},{"text":"NASA reported Webb’s own first direct image","href":"https://science.nasa.gov/blogs/webb/2022/09/01/nasas-webb-takes-its-first-ever-direct-image-of-distant-world/"},{"text":" of an exoplanet. The word “own” changes what the claim means."}]},{"t":"p","runs":[{"text":"The answer is an example of a wrong factual claim. The example does not tell us which part of Bard’s system produced it."}]},{"t":"quick","prompt":"Before the answer: how far was Bard’s sentence from a true one?","options":["Far: the whole claim was invented","About one word","Not at all: it was true then"],"answer":1,"reveal":"About one word. In September 2022 NASA announced Webb’s first direct image of a planet outside our solar system. Drop the word “Webb’s” and you have Bard’s false claim. A system that writes likely text slides across a word like that easily."},{"t":"h2","id":"next-pieces-not-a-list-of-verified-facts","runs":[{"text":"Next pieces, not a list of verified facts"}]},{"t":"p","runs":[{"text":"Many text-generating language models are first trained to predict a next token from previous tokens. Tokens are pieces of text, which may be whole words, parts of words or punctuation. The model computes scores over possible next tokens. A decoding rule chooses one and the process continues. Some systems add other training objectives or steps; this is a useful sketch of a common text generation loop."}]},{"t":"p","runs":[{"text":"Prediction can learn useful patterns about grammar, language and the world. It does not by itself look up an original source or prove that a statement is true. A deployed assistant may also retrieve documents, use tools or run checks. The reliability of those checks depends on whether they ran, what evidence they found and how the assistant used it."}]},{"t":"figure","label":"Figure","figure":{"type":"flow","title":"A common next-token generation loop","caption":"This sketches text generation itself. A deployed assistant may also retrieve sources or use tools. Check whether that happened and whether the evidence supports the answer.","steps":[{"label":"Text context","detail":"The input and text already generated."},{"label":"Represent text as tokens","detail":"Pieces may be words, parts of words or punctuation."},{"label":"Score possible continuations","detail":"The model uses its learned parameters."},{"label":"Choose the next token","detail":"A decoding rule selects from the scores."},{"label":"Continue or stop","detail":"The system can stop at an end signal or at a length limit."}]},"alt":"Figure: A common next-token generation loop. Steps: 1. Text context (The input and text already generated.); 2. Represent text as tokens (Pieces may be words, parts of words or punctuation.); 3. Score possible continuations (The model uses its learned parameters.); 4. Choose the next token (A decoding rule selects from the scores.); 5. Continue or stop (The system can stop at an end signal or at a length limit.). This sketches text generation itself. A deployed assistant may also retrieve sources or use tools. Check whether that happened and whether the evidence supports the answer."},{"t":"h2","id":"where-the-assistant-s-behaviour-comes-from","runs":[{"text":"Where the assistant’s behaviour comes from"}]},{"t":"p","runs":[{"text":"Pretraining","b":true},{"text":" teaches patterns from large amounts of data. The exact training mix and objectives depend on the model."}]},{"t":"p","runs":[{"text":"Instruction tuning","b":true},{"text":" can teach how to respond to requests by learning from examples. "},{"text":"Preference training","b":true},{"text":" can steer the assistant towards responses people prefer; "},{"text":"the InstructGPT paper","href":"https://arxiv.org/abs/2203.02155"},{"text":" describes one method. In a separate set of experiments, "},{"text":"Sharma and colleagues","href":"https://arxiv.org/abs/2310.13548"},{"text":" found that some assistants sometimes agreed with a user’s stated view even when it was wrong. That does not mean every assistant will do so in every setting."}]},{"t":"p","runs":[{"text":"Models can also show signals related to their uncertainty. "},{"text":"Kadavath and colleagues","href":"https://arxiv.org/abs/2207.05221"},{"text":" tested ways to elicit such signals. A confident style in ordinary text is not one of those verified measurements. Treat confidence as something you test under known conditions."}]},{"t":"h2","id":"try-it-open-the-source-before-using-the-fact","runs":[{"text":"Try it: open the source before using the fact"}],"role":"try"},{"t":"p","runs":[{"text":"Write down one factual sentence from a generated explanation. Underline its date, named object and action. Find an independent source that could verify those details. Mark each item “supported,” “contradicted” or “not addressed.” If the source leaves a gap, keep the gap visible. A citation at the end of a paragraph is useful only when it supports the particular claim you are using."}]}],"check":{"question":[{"text":"The same assistant repeats a disputed claim and gives a confident reason. What would add independent evidence?"}],"options":[{"text":[{"text":"Ask the same question twice more."}],"correct":false,"tempting":[{"text":"Repeated answers can repeat the same error."}]},{"text":[{"text":"Open a source that can verify the precise claim."}],"correct":true,"tempting":null},{"text":[{"text":"Accept the claim because it is well phrased."}],"correct":false,"tempting":[{"text":"Fluency and evidence are different things."}]},{"text":[{"text":"Take agreement from a second chatbot as proof."}],"correct":false,"tempting":[{"text":"Two models can inherit the same mistake."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"what-actually-goes-wrong","title":"Design, capability or misuse","minutes":3},"next":{"slug":"training-data-is-a-choice","title":"Where the data came from","minutes":3}}}