{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"tokenizer-failures-in-the-wild","title":"Tokenizer failures you will actually hit","url":"https://addaly.com/learn/how-llms-work/tokenizer-failures-in-the-wild","position":4,"of":78,"minutes":8,"answer":"Several everyday model failures are not reasoning failures at all but artefacts of where the tokenizer happened to cut, and they are diagnosable by printing the split.","keyIdea":"Several everyday model failures are not reasoning failures at all but artefacts of where the tokenizer happened to cut, and they are diagnosable by printing the split."},"body":{"format":"markdown","text":"## A short catalogue of real problems\n\nTokenisation sounds like a preliminary detail. In production it causes a specific, recognisable set of bugs. Here are the ones worth knowing by sight.\n\n## Trailing whitespace breaks completions\n\nSend a completion-style prompt ending in a space (`\"The capital of France is \"`) and quality often drops. The reason is that in normal text, the space belongs to the *front* of the next token: the model expects to produce ` Paris` as one unit, with the leading space included. By supplying the space yourself, you have asked it to produce a token that begins mid-word, which is a rare configuration in training data.\n\nThe fix is trivial once you know: end your prompt at a natural boundary and let the model supply the space. Chat endpoints hide this, but it still bites when you use raw completion or a local model.\n\n## Numbers do not know about place value\n\nDepending on the tokenizer, `1234567` may become `123` + `456` + `7`, or `1` + `234` + `567`, or four fragments split at unhelpful points. Some newer tokenizers deliberately force digits into groups of one or three to make arithmetic more learnable, a design change that measurably improves arithmetic accuracy and tells you how much of the problem was tokenisation in the first place.\n\nPractical consequence: never trust free-form arithmetic on long numbers, and when you must have it, format numbers with separators or hand the calculation to code.\n\n## Glitch tokens\n\nIn 2023 researchers probing GPT-2 and GPT-3 tokenizers found strings such as `SolidGoldMagikarp` that had earned their own token (they were frequent in the tokenizer's training data, often as usernames on scraped forums) but were then largely absent from the model's own training data. The result is a token whose embedding was barely ever updated. Prompting with it produced evasion, insults and non-sequiturs.\n\nThe general lesson survives the specific example: the tokenizer and the model are trained on different data at different times, and any mismatch leaves tokens the model has effectively never seen.\n\n## Code and structured text are token-hungry\n\nIndentation, brackets and repeated keywords tokenise densely. A JSON payload can cost two to three times what the same information costs in prose, because every quotation mark, brace, colon and comma is a token, and long key names split. If you are paying per token or fighting a context limit, changing the format of the data you send is often a bigger win than shortening the prose around it.\n\n## \"Write exactly 100 words\" does not work\n\nThe model has no counter. It cannot see word boundaries directly, and it has no persistent register that increments. What it has is a statistical sense, from training text, of how long a paragraph described as \"brief\" tends to be. So length instructions land as vague style hints. Ask for exactly 100 words and you will get somewhere between 70 and 160.\n\nIf length matters, enforce it outside the model. Generate, count in code, and either truncate or re-ask.\n\n## Case and Unicode\n\n`Bank`, `bank`, ` bank` and `BANK` are different token sequences with different embeddings. So are the ASCII apostrophe and the typographic one, and the several visually identical dashes. A prompt copy-pasted from a word processor can tokenise differently from the same prompt typed in a terminal, which is a maddening bug to chase if you do not know it exists.\n\nNormalising your input (one apostrophe style, one dash style, no non-breaking spaces) is a cheap, boring reliability improvement.\n\n## The diagnostic habit\n\nWhen a model fails on something that looks like it should be easy, print the tokens before you theorise:\n\n```python\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nfor s in [\"strawberry\", \" strawberry\", \"Strawberry\", \"1234567\", \"मुंबई\"]:\n    ids = enc.encode(s)\n    print(repr(s), len(ids), [enc.decode([i]) for i in ids])\n```\n\nTwo minutes with that loop resolves a category of bug that people otherwise attribute to the model being \"bad at details\". The free tokenizer libraries (`tiktoken` for OpenAI encodings, Hugging Face `tokenizers` for nearly everything else) run offline on any laptop and cost nothing.\n\n## Where it stops being the tokenizer\n\nBe honest about the limits of this explanation. Character-level failures, digit splits and length control are tokenisation problems. A model getting a historical date wrong is not. The value of knowing this category is being able to *rule it in or out* in two minutes, rather than adding another sentence to your prompt and hoping."},"check":{"question":"A support tool asks a model to reply with exactly 50 words and it consistently returns between 38 and 71. Which change addresses the actual mechanism?","options":[{"text":"Repeat the instruction at both the start and the end of the prompt, since recall of instructions is position-dependent.","correct":false,"tempting":"Position effects on long prompts are real, which makes this sound like an informed fix, but the model still has no way to count."},{"text":"Lower the temperature so the model follows instructions more strictly.","correct":false,"tempting":"Temperature is the usual dial for \"make it behave\", though it reshapes the distribution rather than adding a counting ability."},{"text":"Give three examples of 50-word replies so the model can learn the target length.","correct":false,"tempting":"Few-shot examples improve style and format, so this is a reasonable instinct; it improves the average without making the count reliable."},{"text":"Generate the reply, count the words in code, and re-ask or truncate, because nothing in the model tracks a running count.","correct":true}]},"prev":{"slug":"building-the-vocabulary","title":"How the vocabulary gets built, by hand","url":"https://addaly.com/learn/how-llms-work/building-the-vocabulary"},"next":{"slug":"meaning-as-direction","title":"Embeddings: meaning as a direction","url":"https://addaly.com/learn/how-llms-work/meaning-as-direction"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Tokenizer failures you will actually hit', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/tokenizer-failures-in-the-wild (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Tokenizer failures you will actually hit","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"Several everyday model failures are not reasoning failures at all but artefacts of where the tokenizer happened to cut, and they are diagnosable by printing the split.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-short-catalogue-of-real-problems","runs":[{"text":"A short catalogue of real problems"}]},{"t":"p","runs":[{"text":"Tokenisation sounds like a preliminary detail. In production it causes a specific, recognisable set of bugs. Here are the ones worth knowing by sight."}]},{"t":"h2","id":"trailing-whitespace-breaks-completions","runs":[{"text":"Trailing whitespace breaks completions"}]},{"t":"p","runs":[{"text":"Send a completion-style prompt ending in a space ("},{"text":"\"The capital of France is \"","code":true},{"text":") and quality often drops. The reason is that in normal text, the space belongs to the "},{"text":"front","i":true},{"text":" of the next token: the model expects to produce "},{"text":" Paris","code":true},{"text":" as one unit, with the leading space included. By supplying the space yourself, you have asked it to produce a token that begins mid-word, which is a rare configuration in training data."}]},{"t":"p","runs":[{"text":"The fix is trivial once you know: end your prompt at a natural boundary and let the model supply the space. Chat endpoints hide this, but it still bites when you use raw completion or a local model."}]},{"t":"h2","id":"numbers-do-not-know-about-place-value","runs":[{"text":"Numbers do not know about place value"}]},{"t":"p","runs":[{"text":"Depending on the tokenizer, "},{"text":"1234567","code":true},{"text":" may become "},{"text":"123","code":true},{"text":" + "},{"text":"456","code":true},{"text":" + "},{"text":"7","code":true},{"text":", or "},{"text":"1","code":true},{"text":" + "},{"text":"234","code":true},{"text":" + "},{"text":"567","code":true},{"text":", or four fragments split at unhelpful points. Some newer tokenizers deliberately force digits into groups of one or three to make arithmetic more learnable, a design change that measurably improves arithmetic accuracy and tells you how much of the problem was tokenisation in the first place."}]},{"t":"p","runs":[{"text":"Practical consequence: never trust free-form arithmetic on long numbers, and when you must have it, format numbers with separators or hand the calculation to code."}]},{"t":"h2","id":"glitch-tokens","runs":[{"text":"Glitch tokens"}]},{"t":"p","runs":[{"text":"In 2023 researchers probing GPT-2 and GPT-3 tokenizers found strings such as "},{"text":"SolidGoldMagikarp","code":true},{"text":" that had earned their own token (they were frequent in the tokenizer’s training data, often as usernames on scraped forums) but were then largely absent from the model’s own training data. The result is a token whose embedding was barely ever updated. Prompting with it produced evasion, insults and non-sequiturs."}]},{"t":"p","runs":[{"text":"The general lesson survives the specific example: the tokenizer and the model are trained on different data at different times, and any mismatch leaves tokens the model has effectively never seen."}]},{"t":"h2","id":"code-and-structured-text-are-token-hungry","runs":[{"text":"Code and structured text are token-hungry"}]},{"t":"p","runs":[{"text":"Indentation, brackets and repeated keywords tokenise densely. A JSON payload can cost two to three times what the same information costs in prose, because every quotation mark, brace, colon and comma is a token, and long key names split. If you are paying per token or fighting a context limit, changing the format of the data you send is often a bigger win than shortening the prose around it."}]},{"t":"h2","id":"write-exactly-100-words-does-not-work","runs":[{"text":"“Write exactly 100 words” does not work"}]},{"t":"p","runs":[{"text":"The model has no counter. It cannot see word boundaries directly, and it has no persistent register that increments. What it has is a statistical sense, from training text, of how long a paragraph described as “brief” tends to be. So length instructions land as vague style hints. Ask for exactly 100 words and you will get somewhere between 70 and 160."}]},{"t":"p","runs":[{"text":"If length matters, enforce it outside the model. Generate, count in code, and either truncate or re-ask."}]},{"t":"h2","id":"case-and-unicode","runs":[{"text":"Case and Unicode"}]},{"t":"p","runs":[{"text":"Bank","code":true},{"text":", "},{"text":"bank","code":true},{"text":", "},{"text":" bank","code":true},{"text":" and "},{"text":"BANK","code":true},{"text":" are different token sequences with different embeddings. So are the ASCII apostrophe and the typographic one, and the several visually identical dashes. A prompt copy-pasted from a word processor can tokenise differently from the same prompt typed in a terminal, which is a maddening bug to chase if you do not know it exists."}]},{"t":"p","runs":[{"text":"Normalising your input (one apostrophe style, one dash style, no non-breaking spaces) is a cheap, boring reliability improvement."}]},{"t":"h2","id":"the-diagnostic-habit","runs":[{"text":"The diagnostic habit"}]},{"t":"p","runs":[{"text":"When a model fails on something that looks like it should be easy, print the tokens before you theorise:"}]},{"t":"code","lang":"python","label":"Python","text":"import tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nfor s in [\"strawberry\", \" strawberry\", \"Strawberry\", \"1234567\", \"मुंबई\"]:\n    ids = enc.encode(s)\n    print(repr(s), len(ids), [enc.decode([i]) for i in ids])"},{"t":"p","runs":[{"text":"Two minutes with that loop resolves a category of bug that people otherwise attribute to the model being “bad at details”. The free tokenizer libraries ("},{"text":"tiktoken","code":true},{"text":" for OpenAI encodings, Hugging Face "},{"text":"tokenizers","code":true},{"text":" for nearly everything else) run offline on any laptop and cost nothing."}]},{"t":"h2","id":"where-it-stops-being-the-tokenizer","runs":[{"text":"Where it stops being the tokenizer"}]},{"t":"p","runs":[{"text":"Be honest about the limits of this explanation. Character-level failures, digit splits and length control are tokenisation problems. A model getting a historical date wrong is not. The value of knowing this category is being able to "},{"text":"rule it in or out","i":true},{"text":" in two minutes, rather than adding another sentence to your prompt and hoping."}]}],"check":{"question":[{"text":"A support tool asks a model to reply with exactly 50 words and it consistently returns between 38 and 71. Which change addresses the actual mechanism?"}],"options":[{"text":[{"text":"Repeat the instruction at both the start and the end of the prompt, since recall of instructions is position-dependent."}],"correct":false,"tempting":[{"text":"Position effects on long prompts are real, which makes this sound like an informed fix, but the model still has no way to count."}]},{"text":[{"text":"Lower the temperature so the model follows instructions more strictly."}],"correct":false,"tempting":[{"text":"Temperature is the usual dial for “make it behave”, though it reshapes the distribution rather than adding a counting ability."}]},{"text":[{"text":"Give three examples of 50-word replies so the model can learn the target length."}],"correct":false,"tempting":[{"text":"Few-shot examples improve style and format, so this is a reasonable instinct; it improves the average without making the count reliable."}]},{"text":[{"text":"Generate the reply, count the words in code, and re-ask or truncate, because nothing in the model tracks a running count."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"building-the-vocabulary","title":"How the vocabulary gets built, by hand","minutes":9},"next":{"slug":"meaning-as-direction","title":"Embeddings: meaning as a direction","minutes":7}}}