{"course":{"slug":"ai-for-small-business","title":"AI for a Small Business","url":"https://addaly.com/learn/ai-for-small-business","level":"beginner","track":"work"},"lesson":{"slug":"what-the-machine-is-doing","title":"Fluent and wrong come from the same place","url":"https://addaly.com/learn/ai-for-small-business/what-the-machine-is-doing","position":2,"of":88,"minutes":10,"answer":"A language model predicts likely next words rather than retrieving facts, so its confidence is a property of the writing and never evidence about truth.","keyIdea":"A language model predicts likely next words rather than retrieving facts, so its confidence is a property of the writing and never evidence about truth, which is why the facts must go in for the words to come out safely."},"body":{"format":"markdown","text":"## Two words at a time\n\nYou do not need the mathematics. You do need one mechanical fact, because everything useful and everything dangerous about these tools follows from it.\n\nA language model reads what is in front of it and predicts the next fragment of text. Then it reads that, and predicts the next. A fragment is called a **token**: roughly three quarters of a word, so a 400-word email is about 530 tokens. It does this somewhere between a few dozen and a couple of hundred times a second, depending on the model, and a paragraph appears.\n\nThat is the whole trick. There is no lookup, no fact store, no moment where it checks anything. It has read an enormous amount of writing and learned which words tend to follow which, in which contexts.\n\nNow ask it what time your shop opens.\n\nIt has never seen your shop. But it has read tens of thousands of pages about shops, and on those pages the sentence \"We are open\" is very often followed by \"from 9am to 6pm, Monday to Saturday.\" So that is what comes out: in your voice, formatted correctly, entirely invented.\n\nThis is not the model malfunctioning. It is the model doing precisely the one thing it does. People call these inventions hallucinations, which makes them sound like a rare glitch. They are the ordinary output of a machine whose job is plausibility.\n\n## Why it cannot warn you\n\nThe obvious question: why does it not say \"I am unsure about that\"?\n\nBecause the number it produces is the probability of the next token, and that is a statement about word choice, not about truth. When it writes \"9am\", the model is confident that \"9am\" is a likely word in that position. Models do carry some internal signal about whether they know something. Researchers have found features that recognise familiar names and switch off a default \"can't answer\" response, and models can be trained to estimate whether they know an answer. But that signal is unreliable: it misfires on half-familiar names, and nothing in a normal chat shows it to you.\n\nSo the confidence in the writing is a property of the writing. A fabricated price arrives in exactly the same calm, well-punctuated sentence as a true one. There is no tell. Anyone who tells you they can spot invented output by how it reads is describing luck.\n\n## Same question, different answer\n\nTry this once, deliberately. Ask the same question three times in three fresh chats. You will get three different replies.\n\nThat is because the model does not always take the single most likely next token; it samples from the likely ones. Some tools expose this as a \"temperature\" or \"creativity\" setting. Turned down, output is repetitive and safe. Turned up, it is varied and more prone to wandering.\n\nThe business consequence is the one people miss: **you cannot test this thing once.** A single good answer tells you almost nothing, because a different one was available. When you evaluate a prompt later in this course, you will run ten cases, not one.\n\n## What it has read, and when it stopped\n\nEvery model has a training cutoff: a date after which it has read nothing. Ask about a tax change from three months ago and you may get a fluent description of the old rule, or a fluent description of a rule that never existed.\n\nSome tools now attach extras: a web search, a calculator, your uploaded files. When those are switched on the picture improves, because the model is reading a real page instead of reaching for a memory. It is worth knowing whether the tool you are using has search on or off, because the same question gives materially different answers either way, and the interface rarely makes it obvious.\n\n## What this predicts about your work\n\nThe mechanism sorts your tasks for you.\n\nIt will be strong wherever the **shape** of the answer is the hard part and the facts come from you: turning your notes into a scope section, rewriting an angry reply calmly, sorting 300 transaction descriptions into categories, producing forty product descriptions in the pattern of five you wrote yourself.\n\nIt will be weak wherever the **content** has to be specific, private, recent, or numeric: your prices, your lead times, this month's regulations, the total of a column, what your supplier said on the phone.\n\nA reasonable rule that follows directly: **the facts go in, the words come out.** If you find yourself hoping a fact will come out, stop. You have just asked a plausibility machine for a matter of record.\n\n## The useful reframe\n\nStop thinking of it as something that knows things. Think of it as an extremely well-read assistant with no memory of your business, no access to your files, no ability to check anything, and an unshakeable reluctance to admit ignorance.\n\nYou would still hire that person. You would just never let them answer a customer unsupervised, and you would give them your price list first."},"check":{"question":"You ask a model, with no documents attached, what your workshop's delivery lead time is. It answers \"typically 7 to 10 working days\" in a confident, well-formed sentence. What has actually happened?","options":[{"text":"It produced the continuation that is most plausible after that question given everything it has read about workshops, which is unrelated to your actual lead time.","correct":true},{"text":"It has retrieved an industry average from its training data, so the figure is a reasonable starting estimate for your business.","correct":false,"tempting":"Treats generation as retrieval; there is no averaged figure stored anywhere, only a likely sequence of words, so the number carries no evidential weight at all."},{"text":"It guessed, and the confident phrasing means the guess sits near the top of its probability range, so the figure is more likely to be right than wrong and can be treated as a working estimate.","correct":false,"tempting":"Reads fluency as a confidence signal, but the probability governs word choice rather than truth, so smooth phrasing and invented content arrive together."},{"text":"The model has malfunctioned and needs a clearer prompt specifying that it should only answer from verified sources.","correct":false,"tempting":"Frames ordinary behaviour as a fault; a clearer instruction can reduce invention, but nothing in the prompt gives it access to a lead time it was never told."}]},"prev":{"slug":"find-the-two-tasks","title":"The Two Tasks Worth Doing","url":"https://addaly.com/learn/ai-for-small-business/find-the-two-tasks"},"next":{"slug":"the-checking-tax","title":"The checking tax nobody counts","url":"https://addaly.com/learn/ai-for-small-business/the-checking-tax"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fluent and wrong come from the same place', in AI for a Small Business. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-small-business/what-the-machine-is-doing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fluent and wrong come from the same place","courseTitle":"AI for a Small Business","module":{"index":1,"slug":"where-it-belongs","title":"Working out where AI belongs in your business"},"answer":null,"keyIdea":"A language model predicts likely next words rather than retrieving facts, so its confidence is a property of the writing and never evidence about truth, which is why the facts must go in for the words to come out safely.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-words-at-a-time","runs":[{"text":"Two words at a time"}]},{"t":"p","runs":[{"text":"You do not need the mathematics. You do need one mechanical fact, because everything useful and everything dangerous about these tools follows from it."}]},{"t":"p","runs":[{"text":"A language model reads what is in front of it and predicts the next fragment of text. Then it reads that, and predicts the next. A fragment is called a "},{"text":"token","b":true},{"text":": roughly three quarters of a word, so a 400-word email is about 530 tokens. It does this somewhere between a few dozen and a couple of hundred times a second, depending on the model, and a paragraph appears."}]},{"t":"p","runs":[{"text":"That is the whole trick. There is no lookup, no fact store, no moment where it checks anything. It has read an enormous amount of writing and learned which words tend to follow which, in which contexts."}]},{"t":"p","runs":[{"text":"Now ask it what time your shop opens."}]},{"t":"p","runs":[{"text":"It has never seen your shop. But it has read tens of thousands of pages about shops, and on those pages the sentence “We are open” is very often followed by “from 9am to 6pm, Monday to Saturday.” So that is what comes out: in your voice, formatted correctly, entirely invented."}]},{"t":"p","runs":[{"text":"This is not the model malfunctioning. It is the model doing precisely the one thing it does. People call these inventions hallucinations, which makes them sound like a rare glitch. They are the ordinary output of a machine whose job is plausibility."}]},{"t":"h2","id":"why-it-cannot-warn-you","runs":[{"text":"Why it cannot warn you"}]},{"t":"p","runs":[{"text":"The obvious question: why does it not say “I am unsure about that”?"}]},{"t":"p","runs":[{"text":"Because the number it produces is the probability of the next token, and that is a statement about word choice, not about truth. When it writes “9am”, the model is confident that “9am” is a likely word in that position. Models do carry some internal signal about whether they know something. Researchers have found features that recognise familiar names and switch off a default “can’t answer” response, and models can be trained to estimate whether they know an answer. But that signal is unreliable: it misfires on half-familiar names, and nothing in a normal chat shows it to you."}]},{"t":"p","runs":[{"text":"So the confidence in the writing is a property of the writing. A fabricated price arrives in exactly the same calm, well-punctuated sentence as a true one. There is no tell. Anyone who tells you they can spot invented output by how it reads is describing luck."}]},{"t":"h2","id":"same-question-different-answer","runs":[{"text":"Same question, different answer"}]},{"t":"p","runs":[{"text":"Try this once, deliberately. Ask the same question three times in three fresh chats. You will get three different replies."}]},{"t":"p","runs":[{"text":"That is because the model does not always take the single most likely next token; it samples from the likely ones. Some tools expose this as a “temperature” or “creativity” setting. Turned down, output is repetitive and safe. Turned up, it is varied and more prone to wandering."}]},{"t":"p","runs":[{"text":"The business consequence is the one people miss: "},{"text":"you cannot test this thing once.","b":true},{"text":" A single good answer tells you almost nothing, because a different one was available. When you evaluate a prompt later in this course, you will run ten cases, not one."}]},{"t":"h2","id":"what-it-has-read-and-when-it-stopped","runs":[{"text":"What it has read, and when it stopped"}]},{"t":"p","runs":[{"text":"Every model has a training cutoff: a date after which it has read nothing. Ask about a tax change from three months ago and you may get a fluent description of the old rule, or a fluent description of a rule that never existed."}]},{"t":"p","runs":[{"text":"Some tools now attach extras: a web search, a calculator, your uploaded files. When those are switched on the picture improves, because the model is reading a real page instead of reaching for a memory. It is worth knowing whether the tool you are using has search on or off, because the same question gives materially different answers either way, and the interface rarely makes it obvious."}]},{"t":"h2","id":"what-this-predicts-about-your-work","runs":[{"text":"What this predicts about your work"}]},{"t":"p","runs":[{"text":"The mechanism sorts your tasks for you."}]},{"t":"p","runs":[{"text":"It will be strong wherever the "},{"text":"shape","b":true},{"text":" of the answer is the hard part and the facts come from you: turning your notes into a scope section, rewriting an angry reply calmly, sorting 300 transaction descriptions into categories, producing forty product descriptions in the pattern of five you wrote yourself."}]},{"t":"p","runs":[{"text":"It will be weak wherever the "},{"text":"content","b":true},{"text":" has to be specific, private, recent, or numeric: your prices, your lead times, this month’s regulations, the total of a column, what your supplier said on the phone."}]},{"t":"p","runs":[{"text":"A reasonable rule that follows directly: "},{"text":"the facts go in, the words come out.","b":true},{"text":" If you find yourself hoping a fact will come out, stop. You have just asked a plausibility machine for a matter of record."}]},{"t":"h2","id":"the-useful-reframe","runs":[{"text":"The useful reframe"}]},{"t":"p","runs":[{"text":"Stop thinking of it as something that knows things. Think of it as an extremely well-read assistant with no memory of your business, no access to your files, no ability to check anything, and an unshakeable reluctance to admit ignorance."}]},{"t":"p","runs":[{"text":"You would still hire that person. You would just never let them answer a customer unsupervised, and you would give them your price list first."}]}],"check":{"question":[{"text":"You ask a model, with no documents attached, what your workshop’s delivery lead time is. It answers “typically 7 to 10 working days” in a confident, well-formed sentence. What has actually happened?"}],"options":[{"text":[{"text":"It produced the continuation that is most plausible after that question given everything it has read about workshops, which is unrelated to your actual lead time."}],"correct":true,"tempting":null},{"text":[{"text":"It has retrieved an industry average from its training data, so the figure is a reasonable starting estimate for your business."}],"correct":false,"tempting":[{"text":"Treats generation as retrieval; there is no averaged figure stored anywhere, only a likely sequence of words, so the number carries no evidential weight at all."}]},{"text":[{"text":"It guessed, and the confident phrasing means the guess sits near the top of its probability range, so the figure is more likely to be right than wrong and can be treated as a working estimate."}],"correct":false,"tempting":[{"text":"Reads fluency as a confidence signal, but the probability governs word choice rather than truth, so smooth phrasing and invented content arrive together."}]},{"text":[{"text":"The model has malfunctioned and needs a clearer prompt specifying that it should only answer from verified sources."}],"correct":false,"tempting":[{"text":"Frames ordinary behaviour as a fault; a clearer instruction can reduce invention, but nothing in the prompt gives it access to a lead time it was never told."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"find-the-two-tasks","title":"The Two Tasks Worth Doing","minutes":7},"next":{"slug":"the-checking-tax","title":"The checking tax nobody counts","minutes":9}}}