{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"tokens-not-words","title":"Tokens, and why a token is not a word","url":"https://addaly.com/learn/how-llms-work/tokens-not-words","position":2,"of":78,"minutes":8,"answer":"A token is a chunk of characters chosen by a compression algorithm, not a word and not a letter.","keyIdea":"A token is a chunk of characters chosen by a compression algorithm, not a word and not a letter."},"body":{"format":"markdown","text":"## Your text becomes numbers before anything else happens\n\nWhen you type \"Where is the nearest chemist?\" the model does not receive letters. A tokenizer cuts the string into pieces drawn from a fixed vocabulary (usually 50,000 to 200,000 entries) and hands over a list of integers. Everything after that point is arithmetic on those integers. The model has no other access to what you wrote.\n\nThat vocabulary is not a dictionary. It is built by an algorithm, usually byte pair encoding, which starts with single bytes and repeatedly merges whichever adjacent pair is most frequent in a training corpus. Sequences that appear constantly end up as one token. Rare ones stay in pieces.\n\n## What follows from that\n\n- Common short words are one token each: \" the\", \" and\", \" is\". Note the leading space. It is part of the token.\n- A long common word is often one token. An unusual one is several. A surname, a Sanskrit term, or a chemical name may cost six.\n- Capitalisation and spacing change the token. \"bank\", \" bank\", \"Bank\" and \" Bank\" are four different integers.\n- Numbers split in ways unrelated to place value. GPT-2's tokenizer has 2016 as a single token but splits 20164 into \"201\" plus \"64\"; GPT-4's cuts a run of digits into groups of three from the left, so 2026 becomes \"202\" plus \"6\" and 20264 becomes \"202\" plus \"64\"; the original Llama tokenizer gives every digit its own token.\n\n## English is cheaper than Hindi, and that is a product decision\n\nTokenizer vocabularies are built from corpora dominated by English, so English averages roughly four characters per token. On GPT-4's tokenizer (cl100k_base), Hindi and Thai land near one token per character and Tamil, Telugu, Amharic and Burmese need more than one, so the same sentence costs about 4 to 12 times as many tokens as in English. GPT-4o's larger vocabulary (o200k_base) cuts most of these to 1.5 to 3 times, though Amharic stays near 6. Measure your own.\n\nTokens are both the billing unit and the context unit, so this is not a curiosity. A support transcript in Hindi can cost several times more to process than the same transcript in English and eat several times more of the context window. If you are choosing a model for a multilingual product, count tokens per language before you compare anything else. Prices quoted per million tokens are not comparable across languages.\n\n> Figure: The same paragraph, five ways of writing it. Bars (tokens for one 60-word paragraph): English 78; Romanised Hinglish 111; Hindi, in Devanagari 172; Bengali 181; Tamil 206. Measured on a 2024-generation tokenizer with a 128,000-entry vocabulary. Tokens are both the bill and the context unit, so the same support conversation costs a Hindi user about twice what it costs an English one; and the older 50,000-entry tokenizers were three to eight times worse.\n\n## The strawberry problem\n\nAsk a model how many times \"r\" appears in \"strawberry\" and it may say two. This is usually described as a reasoning failure. It is closer to a visibility failure.\n\nThe word arrives as something like \"str\" + \"aw\" + \"berry\": three integers. Nothing in that input exposes individual characters. Whatever the model knows about the spelling of the word, it absorbed indirectly, from text that discussed spelling, hyphenation, rhyme or wordplay. It is second-hand knowledge about a thing it cannot see.\n\nThe same explanation covers reversing strings, counting characters, and judging syllables. Meanwhile the model handles far harder semantic work without trouble, which is why the failure feels so strange. If you need character-level work, hand it to code:\n\n```python\ntext = \"strawberry\"\nprint(text.count(\"r\"))   # 3, every time\n```\n\n## Look at your own splits\n\nEvery major provider ships a tokenizer you can run locally. Ten minutes with it changes how you write prompts.\n\n```python\nimport tiktoken\nenc = tiktoken.get_encoding(\"cl100k_base\")\nids = enc.encode(\"Where is the nearest chemist?\")\nprint(len(ids), [enc.decode([i]) for i in ids])\n```\n\nRun it on a prompt template you use daily. Then run it on the same template translated into a language your users actually speak. The difference is usually larger than people expect.\n\n## Why subwords at all\n\nA word-level vocabulary cannot represent anything it has not seen: every new name, typo or product code becomes a single \"unknown\" token, and the information is gone. A character-level vocabulary handles everything but makes sequences four to five times longer, and the cost of attention grows with the square of length, which you will meet in lesson three.\n\nSubword tokens are the compromise. Any string can be represented, because the fallback is raw bytes, and ordinary text stays short. The price is that the model's view of language is chunked in a way that follows statistics, not meaning or spelling."},"check":{"question":"A model is asked how many times the letter \"r\" appears in \"strawberry\" and answers two. It handles a request to summarise a legal paragraph correctly. What best explains the difference?","options":[{"text":"Counting is arithmetic, and arithmetic is the known weak spot of these models.","correct":false,"tempting":"Counting to three does look like a maths problem, and everyone has heard that models are unreliable at maths."},{"text":"The tokenizer compresses the word and drops the repeated letters along the way.","correct":false,"tempting":"\"Compression\" sounds like information is being thrown away, which would neatly explain a missing letter."},{"text":"The word arrives as two or three tokens, so the individual letters are never separately present in the input; whatever the model knows about its spelling is second-hand.","correct":true},{"text":"There is a gap in the training data: very few documents discuss the spelling of \"strawberry\".","correct":false,"tempting":"Blaming training coverage explains many real failures, so it is a habit that usually pays off."}]},"prev":{"slug":"what-a-language-model-is","title":"What a language model actually is","url":"https://addaly.com/learn/how-llms-work/what-a-language-model-is"},"next":{"slug":"building-the-vocabulary","title":"How the vocabulary gets built, by hand","url":"https://addaly.com/learn/how-llms-work/building-the-vocabulary"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Tokens, and why a token is not a word', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/tokens-not-words (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Tokens, and why a token is not a word","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"A token is a chunk of characters chosen by a compression algorithm, not a word and not a letter.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens for one 60-word paragraph","title":"The same paragraph, five ways of writing it","series":[{"label":"English","value":78},{"label":"Romanised Hinglish","value":111},{"label":"Hindi, in Devanagari","value":172,"highlight":true},{"label":"Bengali","value":181},{"label":"Tamil","value":206}],"caption":"Measured on a 2024-generation tokenizer with a 128,000-entry vocabulary. Tokens are both the bill and the context unit, so the same support conversation costs a Hindi user about twice what it costs an English one; and the older 50,000-entry tokenizers were three to eight times worse."},"alt":"Figure: The same paragraph, five ways of writing it. Bars (tokens for one 60-word paragraph): English 78; Romanised Hinglish 111; Hindi, in Devanagari 172; Bengali 181; Tamil 206. Measured on a 2024-generation tokenizer with a 128,000-entry vocabulary. Tokens are both the bill and the context unit, so the same support conversation costs a Hindi user about twice what it costs an English one; and the older 50,000-entry tokenizers were three to eight times worse."}},"blocks":[{"t":"h2","id":"your-text-becomes-numbers-before-anything-else-happens","runs":[{"text":"Your text becomes numbers before anything else happens"}]},{"t":"p","runs":[{"text":"When you type “Where is the nearest chemist?” the model does not receive letters. A tokenizer cuts the string into pieces drawn from a fixed vocabulary (usually 50,000 to 200,000 entries) and hands over a list of integers. Everything after that point is arithmetic on those integers. The model has no other access to what you wrote."}]},{"t":"p","runs":[{"text":"That vocabulary is not a dictionary. It is built by an algorithm, usually byte pair encoding, which starts with single bytes and repeatedly merges whichever adjacent pair is most frequent in a training corpus. Sequences that appear constantly end up as one token. Rare ones stay in pieces."}]},{"t":"h2","id":"what-follows-from-that","runs":[{"text":"What follows from that"}]},{"t":"ul","items":[[{"text":"Common short words are one token each: “ the”, “ and”, “ is”. Note the leading space. It is part of the token."}],[{"text":"A long common word is often one token. An unusual one is several. A surname, a Sanskrit term, or a chemical name may cost six."}],[{"text":"Capitalisation and spacing change the token. “bank”, “ bank”, “Bank” and “ Bank” are four different integers."}],[{"text":"Numbers split in ways unrelated to place value. GPT-2’s tokenizer has 2016 as a single token but splits 20164 into “201” plus “64”; GPT-4’s cuts a run of digits into groups of three from the left, so 2026 becomes “202” plus “6” and 20264 becomes “202” plus “64”; the original Llama tokenizer gives every digit its own token."}]]},{"t":"h2","id":"english-is-cheaper-than-hindi-and-that-is-a-product-decision","runs":[{"text":"English is cheaper than Hindi, and that is a product decision"}]},{"t":"p","runs":[{"text":"Tokenizer vocabularies are built from corpora dominated by English, so English averages roughly four characters per token. On GPT-4’s tokenizer (cl100k_base), Hindi and Thai land near one token per character and Tamil, Telugu, Amharic and Burmese need more than one, so the same sentence costs about 4 to 12 times as many tokens as in English. GPT-4o’s larger vocabulary (o200k_base) cuts most of these to 1.5 to 3 times, though Amharic stays near 6. Measure your own."}]},{"t":"p","runs":[{"text":"Tokens are both the billing unit and the context unit, so this is not a curiosity. A support transcript in Hindi can cost several times more to process than the same transcript in English and eat several times more of the context window. If you are choosing a model for a multilingual product, count tokens per language before you compare anything else. Prices quoted per million tokens are not comparable across languages."}]},{"t":"h2","id":"the-strawberry-problem","runs":[{"text":"The strawberry problem"}]},{"t":"p","runs":[{"text":"Ask a model how many times “r” appears in “strawberry” and it may say two. This is usually described as a reasoning failure. It is closer to a visibility failure."}]},{"t":"p","runs":[{"text":"The word arrives as something like “str” + “aw” + “berry”: three integers. Nothing in that input exposes individual characters. Whatever the model knows about the spelling of the word, it absorbed indirectly, from text that discussed spelling, hyphenation, rhyme or wordplay. It is second-hand knowledge about a thing it cannot see."}]},{"t":"p","runs":[{"text":"The same explanation covers reversing strings, counting characters, and judging syllables. Meanwhile the model handles far harder semantic work without trouble, which is why the failure feels so strange. If you need character-level work, hand it to code:"}]},{"t":"code","lang":"python","label":"Python","text":"text = \"strawberry\"\nprint(text.count(\"r\"))   # 3, every time"},{"t":"h2","id":"look-at-your-own-splits","runs":[{"text":"Look at your own splits"}]},{"t":"p","runs":[{"text":"Every major provider ships a tokenizer you can run locally. Ten minutes with it changes how you write prompts."}]},{"t":"code","lang":"python","label":"Python","text":"import tiktoken\nenc = tiktoken.get_encoding(\"cl100k_base\")\nids = enc.encode(\"Where is the nearest chemist?\")\nprint(len(ids), [enc.decode([i]) for i in ids])"},{"t":"p","runs":[{"text":"Run it on a prompt template you use daily. Then run it on the same template translated into a language your users actually speak. The difference is usually larger than people expect."}]},{"t":"h2","id":"why-subwords-at-all","runs":[{"text":"Why subwords at all"}]},{"t":"p","runs":[{"text":"A word-level vocabulary cannot represent anything it has not seen: every new name, typo or product code becomes a single “unknown” token, and the information is gone. A character-level vocabulary handles everything but makes sequences four to five times longer, and the cost of attention grows with the square of length, which you will meet in lesson three."}]},{"t":"p","runs":[{"text":"Subword tokens are the compromise. Any string can be represented, because the fallback is raw bytes, and ordinary text stays short. The price is that the model’s view of language is chunked in a way that follows statistics, not meaning or spelling."}]}],"check":{"question":[{"text":"A model is asked how many times the letter “r” appears in “strawberry” and answers two. It handles a request to summarise a legal paragraph correctly. What best explains the difference?"}],"options":[{"text":[{"text":"Counting is arithmetic, and arithmetic is the known weak spot of these models."}],"correct":false,"tempting":[{"text":"Counting to three does look like a maths problem, and everyone has heard that models are unreliable at maths."}]},{"text":[{"text":"The tokenizer compresses the word and drops the repeated letters along the way."}],"correct":false,"tempting":[{"text":"“Compression” sounds like information is being thrown away, which would neatly explain a missing letter."}]},{"text":[{"text":"The word arrives as two or three tokens, so the individual letters are never separately present in the input; whatever the model knows about its spelling is second-hand."}],"correct":true,"tempting":null},{"text":[{"text":"There is a gap in the training data: very few documents discuss the spelling of “strawberry”."}],"correct":false,"tempting":[{"text":"Blaming training coverage explains many real failures, so it is a habit that usually pays off."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"what-a-language-model-is","title":"What a language model actually is","minutes":8},"next":{"slug":"building-the-vocabulary","title":"How the vocabulary gets built, by hand","minutes":9}}}