Tokens, and why a token is not a word

Your text becomes numbers before anything else happens

When you type “Where is the nearest chemist?” the model does not receive letters. A tokenizer cuts the string into pieces drawn from a fixed vocabulary (usually 50,000 to 200,000 entries) and hands over a list of integers. Everything after that point is arithmetic on those integers. The model has no other access to what you wrote.

That vocabulary is not a dictionary. It is built by an algorithm, usually byte pair encoding, which starts with single bytes and repeatedly merges whichever adjacent pair is most frequent in a training corpus. Sequences that appear constantly end up as one token. Rare ones stay in pieces.

What follows from that

• Common short words are one token each: “ the”, “ and”, “ is”. Note the leading space. It is part of the token.
• A long common word is often one token. An unusual one is several. A surname, a Sanskrit term, or a chemical name may cost six.
• Capitalisation and spacing change the token. “bank”, “ bank”, “Bank” and “ Bank” are four different integers.
• Numbers split in ways unrelated to place value. GPT-2’s tokenizer has 2016 as a single token but splits 20164 into “201” plus “64”; GPT-4’s cuts a run of digits into groups of three from the left, so 2026 becomes “202” plus “6” and 20264 becomes “202” plus “64”; the original Llama tokenizer gives every digit its own token.

English is cheaper than Hindi, and that is a product decision

Tokenizer vocabularies are built from corpora dominated by English, so English averages roughly four characters per token. On GPT-4’s tokenizer (cl100k_base), Hindi and Thai land near one token per character and Tamil, Telugu, Amharic and Burmese need more than one, so the same sentence costs about 4 to 12 times as many tokens as in English. GPT-4o’s larger vocabulary (o200k_base) cuts most of these to 1.5 to 3 times, though Amharic stays near 6. Measure your own.

Tokens are both the billing unit and the context unit, so this is not a curiosity. A support transcript in Hindi can cost several times more to process than the same transcript in English and eat several times more of the context window. If you are choosing a model for a multilingual product, count tokens per language before you compare anything else. Prices quoted per million tokens are not comparable across languages.

The strawberry problem

Ask a model how many times “r” appears in “strawberry” and it may say two. This is usually described as a reasoning failure. It is closer to a visibility failure.

The word arrives as something like “str” + “aw” + “berry”: three integers. Nothing in that input exposes individual characters. Whatever the model knows about the spelling of the word, it absorbed indirectly, from text that discussed spelling, hyphenation, rhyme or wordplay. It is second-hand knowledge about a thing it cannot see.

The same explanation covers reversing strings, counting characters, and judging syllables. Meanwhile the model handles far harder semantic work without trouble, which is why the failure feels so strange. If you need character-level work, hand it to code:

text = "strawberry"
print(text.count("r"))   # 3, every time

Look at your own splits

Every major provider ships a tokenizer you can run locally. Ten minutes with it changes how you write prompts.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Where is the nearest chemist?")
print(len(ids), [enc.decode([i]) for i in ids])

Run it on a prompt template you use daily. Then run it on the same template translated into a language your users actually speak. The difference is usually larger than people expect.

Why subwords at all

A word-level vocabulary cannot represent anything it has not seen: every new name, typo or product code becomes a single “unknown” token, and the information is gone. A character-level vocabulary handles everything but makes sequences four to five times longer, and the cost of attention grows with the square of length, which you will meet in lesson three.

Subword tokens are the compromise. Any string can be represented, because the fallback is raw bytes, and ordinary text stays short. The price is that the model’s view of language is chunked in a way that follows statistics, not meaning or spelling.