What it is actually doing

One token at a time

A language model does one thing. Given the text so far, it produces a probability for every possible next chunk of text, and one chunk is chosen. Then it does it again, with that chunk added to the text. Then again. An answer that took you fifteen seconds to read was built in a few hundred of these steps, each of which knew nothing about the ones still to come.

The chunks are called tokens. A token is roughly four characters of English, so about three quarters of a word. Common words are single tokens; unusual ones split. understanding is usually one token. Rutherfordium is four or five. A Hindi or Tamil sentence often costs two to three times as many tokens as the same sentence in English, which is why some tools feel slower and hit limits faster in those languages. Nobody chose that as policy. It falls out of how the vocabulary was built, mostly from English text.

You can see the split for yourself. OpenAI’s tokenizer page and the free tiktoken Python library both show it; so does Hugging Face’s tokenizer playground, which is free and needs no account.

Why it cannot count the letters in a word

Ask a model how many times the letter r appears in strawberry and there is a decent chance of a wrong answer, even from a model that will happily discuss protein folding. This is not stupidity. The model never saw the letters. It saw the tokens str, aw, berry. Asking it to count characters is like asking you to count the pen strokes in a word somebody read aloud to you.

This matters more than it looks. The same blindness affects counting words in your essay, alphabetising a list, working with the digits of a long number, and any puzzle that turns on spelling. If you need those, use something that operates on characters: a word counter, a spreadsheet, wc -w in a terminal, or the model’s code tool if it has one, which writes a Python program and runs it.

Where the knowledge sits

There is no database inside. During training, the model adjusted billions of numerical weights so that its next-token predictions matched an enormous quantity of text. What survives is a compressed statistical impression of that text. Facts repeated in thousands of places (the boiling point of water, the year the Second World War ended) are represented strongly and come out reliably. Facts stated once, in one obscure page, are represented weakly or not at all, and the model has no way to tell the difference between recalling one and constructing something that looks like one.

That is why it is excellent on the material in the first two chapters of your textbook and unreliable on the specific figure from the specific 2019 paper your lecturer mentioned.

Temperature, and why the same question gives different answers

At each step the model has a probability distribution over tokens. Temperature controls how much it favours the most likely one. At temperature 0 it takes the top token every time and the same prompt gives nearly the same answer. Higher temperature samples more widely and produces more variety.

Most chat interfaces run somewhere in the middle, which is why asking the same question twice can give you two different answers, and occasionally one right and one wrong. This is worth exploiting: if you are unsure about an answer, open a fresh conversation and ask again in different words. Two independent agreeing answers are weak evidence. Two disagreeing answers are strong evidence that you need a real source.

What it does not have

• No memory between conversations, unless a memory feature is switched on. A new chat starts blank.
• No sense of time. Its training data stops at some date. It may not know that, and will often answer questions about last month as confidently as questions about last century.
• No access to your file unless you paste it or upload it.
• No internal signal for truth. It has a signal for plausible continuation. Those coincide often enough to be useful and diverge often enough to be dangerous.

The practical consequence

Every reliable technique in this course follows from this one mechanism. You ground it in text you supply, because supplied text is stronger than compressed memory. You ask for working, because a chain you can inspect is checkable and a conclusion is not. You verify anything specific, because specific things are the weakly-represented ones. You never let it do the counting.

A model is a fluent, widely-read, extremely fast colleague with no memory, no source list, and no ability to tell you when it is guessing. Treat it exactly like that and most of the trouble disappears.