Text is chopped up before the model sees it

The model never sees letters

Before anything happens, your text is cut into pieces called tokens, and each token is swapped for a number. The model works entirely with those numbers. It has no access to the letters underneath.

A token is usually a common word, a word fragment, or a piece of punctuation. In English the rough rule is four characters to a token, or about 0.75 words. So 750 words of English is roughly 1,000 tokens, and a 300-page book is somewhere near 100,000.

Common words get a token to themselves. Rarer ones are assembled from parts. “Unbelievable” typically becomes something like un + believ + able. A name the tokeniser has not met may fragment into four or five pieces. The vocabulary is fixed when the tokeniser is built (often 50,000 to 200,000 entries) and everything ever written must be expressible from that set.

You can watch this happen yourself. The tiktoken library and Hugging Face’s tokenizers are free and open source, and both OpenAI and Hugging Face host web pages where you paste text and see the pieces. It takes two minutes and it makes the rest of this lesson concrete.

Why this explains so many oddities

Once you know the model sees straw + berry rather than s-t-r-a-w-b-e-r-r-y, several famous failures stop being mysterious.

Asking how many times the letter r appears in “strawberry” asks the model to inspect something it cannot see. It has to reason about the spelling of a word from what it has read about spellings, rather than by looking. That it often gets it right is the surprising part.

The same applies to counting characters, reversing a word, finding words that rhyme in some languages, and doing arithmetic on long numbers: a number such as 4,871,203 may be cut into three or four tokens in a way that has nothing to do with place value, so column arithmetic is not available to it.

The part that is unfair

Here is the fact that matters most to readers outside the English-speaking world, and it is rarely mentioned.

Tokenisers are built from a corpus, and that corpus is overwhelmingly English. A word represented as one token in English commonly needs three, four or five in Hindi, Tamil, Bengali, Telugu, Amharic or Burmese written in their own scripts. A sentence that costs 10 tokens in English can cost 30 to 50 in Devanagari.

That single fact has three consequences, all of them bad, and all of them structural rather than accidental:

• You pay more. Paid services bill per token, so the same meaning costs several times as much in your language.
• You fit less. A context window measured in tokens holds far fewer Hindi words than English ones, so long documents overflow sooner.
• Quality is lower. Fragmenting a word into pieces that do not correspond to its meaningful parts makes the model’s job harder, and this shows up in measured performance.

It has improved. Newer tokenisers use much larger vocabularies: Llama 3 moved to 128,000 entries, and recent OpenAI and Google tokenisers made specific gains on Indian and East Asian languages. The gap has narrowed. It has not closed, and it will not close while the training text remains as skewed as it is.

If you write in one of these languages, this is worth knowing for a practical reason: when a task fails in your language and succeeds in English, the model’s understanding is only one candidate explanation. The other is that your sentence arrived in forty badly chosen fragments.

What to do with this

Count tokens rather than words when you are near a limit or a budget. The tools above are free, and the difference between your estimate and reality is often a factor of three.

Do not ask for character-level work. Counting letters, spelling backwards, strict character limits: hand those to a computer that can see characters. Two lines of Python will count occurrences of a letter perfectly, every time, at no cost. This is a good general habit: give the deterministic part to a program and the judgement part to the model.

Expect names and technical terms to fragment. A model is likelier to garble an unusual proper noun than a common one, and now you know why: it has never seen that sequence of fragments together often enough to hold it steadily.