Tokenizer failures you will actually hit

A short catalogue of real problems

Tokenisation sounds like a preliminary detail. In production it causes a specific, recognisable set of bugs. Here are the ones worth knowing by sight.

Trailing whitespace breaks completions

Send a completion-style prompt ending in a space ("The capital of France is ") and quality often drops. The reason is that in normal text, the space belongs to the front of the next token: the model expects to produce Paris as one unit, with the leading space included. By supplying the space yourself, you have asked it to produce a token that begins mid-word, which is a rare configuration in training data.

The fix is trivial once you know: end your prompt at a natural boundary and let the model supply the space. Chat endpoints hide this, but it still bites when you use raw completion or a local model.

Numbers do not know about place value

Depending on the tokenizer, 1234567 may become 123 + 456 + 7, or 1 + 234 + 567, or four fragments split at unhelpful points. Some newer tokenizers deliberately force digits into groups of one or three to make arithmetic more learnable, a design change that measurably improves arithmetic accuracy and tells you how much of the problem was tokenisation in the first place.

Practical consequence: never trust free-form arithmetic on long numbers, and when you must have it, format numbers with separators or hand the calculation to code.

Glitch tokens

In 2023 researchers probing GPT-2 and GPT-3 tokenizers found strings such as SolidGoldMagikarp that had earned their own token (they were frequent in the tokenizer’s training data, often as usernames on scraped forums) but were then largely absent from the model’s own training data. The result is a token whose embedding was barely ever updated. Prompting with it produced evasion, insults and non-sequiturs.

The general lesson survives the specific example: the tokenizer and the model are trained on different data at different times, and any mismatch leaves tokens the model has effectively never seen.

Code and structured text are token-hungry

Indentation, brackets and repeated keywords tokenise densely. A JSON payload can cost two to three times what the same information costs in prose, because every quotation mark, brace, colon and comma is a token, and long key names split. If you are paying per token or fighting a context limit, changing the format of the data you send is often a bigger win than shortening the prose around it.

“Write exactly 100 words” does not work

The model has no counter. It cannot see word boundaries directly, and it has no persistent register that increments. What it has is a statistical sense, from training text, of how long a paragraph described as “brief” tends to be. So length instructions land as vague style hints. Ask for exactly 100 words and you will get somewhere between 70 and 160.

If length matters, enforce it outside the model. Generate, count in code, and either truncate or re-ask.

Case and Unicode

Bank, bank, bank and BANK are different token sequences with different embeddings. So are the ASCII apostrophe and the typographic one, and the several visually identical dashes. A prompt copy-pasted from a word processor can tokenise differently from the same prompt typed in a terminal, which is a maddening bug to chase if you do not know it exists.

Normalising your input (one apostrophe style, one dash style, no non-breaking spaces) is a cheap, boring reliability improvement.

The diagnostic habit

When a model fails on something that looks like it should be easy, print the tokens before you theorise:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
for s in ["strawberry", " strawberry", "Strawberry", "1234567", "मुंबई"]:
    ids = enc.encode(s)
    print(repr(s), len(ids), [enc.decode([i]) for i in ids])

Two minutes with that loop resolves a category of bug that people otherwise attribute to the model being “bad at details”. The free tokenizer libraries (tiktoken for OpenAI encodings, Hugging Face tokenizers for nearly everything else) run offline on any laptop and cost nothing.

Where it stops being the tokenizer

Be honest about the limits of this explanation. Character-level failures, digit splits and length control are tokenisation problems. A model getting a historical date wrong is not. The value of knowing this category is being able to rule it in or out in two minutes, rather than adding another sentence to your prompt and hoping.