Fluent and wrong come from the same place

Two words at a time

You do not need the mathematics. You do need one mechanical fact, because everything useful and everything dangerous about these tools follows from it.

A language model reads what is in front of it and predicts the next fragment of text. Then it reads that, and predicts the next. A fragment is called a token: roughly three quarters of a word, so a 400-word email is about 530 tokens. It does this somewhere between a few dozen and a couple of hundred times a second, depending on the model, and a paragraph appears.

That is the whole trick. There is no lookup, no fact store, no moment where it checks anything. It has read an enormous amount of writing and learned which words tend to follow which, in which contexts.

Now ask it what time your shop opens.

It has never seen your shop. But it has read tens of thousands of pages about shops, and on those pages the sentence “We are open” is very often followed by “from 9am to 6pm, Monday to Saturday.” So that is what comes out: in your voice, formatted correctly, entirely invented.

This is not the model malfunctioning. It is the model doing precisely the one thing it does. People call these inventions hallucinations, which makes them sound like a rare glitch. They are the ordinary output of a machine whose job is plausibility.

Why it cannot warn you

The obvious question: why does it not say “I am unsure about that”?

Because the number it produces is the probability of the next token, and that is a statement about word choice, not about truth. When it writes “9am”, the model is confident that “9am” is a likely word in that position. Models do carry some internal signal about whether they know something. Researchers have found features that recognise familiar names and switch off a default “can’t answer” response, and models can be trained to estimate whether they know an answer. But that signal is unreliable: it misfires on half-familiar names, and nothing in a normal chat shows it to you.

So the confidence in the writing is a property of the writing. A fabricated price arrives in exactly the same calm, well-punctuated sentence as a true one. There is no tell. Anyone who tells you they can spot invented output by how it reads is describing luck.

Same question, different answer

Try this once, deliberately. Ask the same question three times in three fresh chats. You will get three different replies.

That is because the model does not always take the single most likely next token; it samples from the likely ones. Some tools expose this as a “temperature” or “creativity” setting. Turned down, output is repetitive and safe. Turned up, it is varied and more prone to wandering.

The business consequence is the one people miss: you cannot test this thing once. A single good answer tells you almost nothing, because a different one was available. When you evaluate a prompt later in this course, you will run ten cases, not one.

What it has read, and when it stopped

Every model has a training cutoff: a date after which it has read nothing. Ask about a tax change from three months ago and you may get a fluent description of the old rule, or a fluent description of a rule that never existed.

Some tools now attach extras: a web search, a calculator, your uploaded files. When those are switched on the picture improves, because the model is reading a real page instead of reaching for a memory. It is worth knowing whether the tool you are using has search on or off, because the same question gives materially different answers either way, and the interface rarely makes it obvious.

What this predicts about your work

The mechanism sorts your tasks for you.

It will be strong wherever the shape of the answer is the hard part and the facts come from you: turning your notes into a scope section, rewriting an angry reply calmly, sorting 300 transaction descriptions into categories, producing forty product descriptions in the pattern of five you wrote yourself.

It will be weak wherever the content has to be specific, private, recent, or numeric: your prices, your lead times, this month’s regulations, the total of a column, what your supplier said on the phone.

A reasonable rule that follows directly: the facts go in, the words come out. If you find yourself hoping a fact will come out, stop. You have just asked a plausibility machine for a matter of record.

The useful reframe

Stop thinking of it as something that knows things. Think of it as an extremely well-read assistant with no memory of your business, no access to your files, no ability to check anything, and an unshakeable reluctance to admit ignorance.

You would still hire that person. You would just never let them answer a customer unsupervised, and you would give them your price list first.