{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"building-the-vocabulary","title":"How the vocabulary gets built, by hand","url":"https://addaly.com/learn/how-llms-work/building-the-vocabulary","position":3,"of":78,"minutes":9,"answer":"A tokenizer's vocabulary is the frozen output of a compression run over one corpus, so what it splits cleanly is a fact about that corpus and not about language.","keyIdea":"A tokenizer's vocabulary is the frozen output of a compression run over one corpus, so what it splits cleanly is a fact about that corpus and not about language."},"body":{"format":"markdown","text":"## Merge the commonest pair, repeat\n\nThe previous lesson said the vocabulary is built by byte pair encoding. Here is the algorithm in full, because it takes five minutes and it removes all the mystery.\n\nStart with a corpus and a vocabulary containing every single byte: 256 entries, which guarantees any text can be represented. Then loop:\n\n1. Count every adjacent pair of symbols in the corpus.\n2. Find the most frequent pair.\n3. Add it to the vocabulary as a new symbol, and replace every occurrence.\n4. Stop when the vocabulary reaches the size you wanted.\n\nRun it on a tiny corpus by hand. Take `low low low lower lowest`. Split into characters. The pair `l o` occurs five times, more than any other, so it merges into `lo`. Now `lo w` occurs five times, so it merges into `low`. Then `low ·` with the following space, and `est`, and so on. After a handful of merges, `low` is one symbol and `est` is one symbol, and `lowest` costs two tokens.\n\nNothing in that procedure knows what a word is. It knows what is frequent.\n\n```python\nfrom collections import Counter\n\ndef merges(words, n):\n    # words: {\"l o w </w>\": 5, \"l o w e r </w>\": 2, ...}\n    for _ in range(n):\n        pairs = Counter()\n        for w, freq in words.items():\n            s = w.split()\n            for a, b in zip(s, s[1:]):\n                pairs[(a, b)] += freq\n        if not pairs:\n            break\n        best = max(pairs, key=pairs.get)\n        joined = \"\".join(best)\n        words = {w.replace(\" \".join(best), joined): f for w, f in words.items()}\n        print(\"merged\", best, \"->\", joined)\n    return words\n```\n\nThirty lines gets you the real thing. The production versions add byte-level fallback, a regular expression that stops merges from crossing certain boundaries, and speed.\n\n## What that means for you\n\nThree consequences follow directly, and each one shows up in practice.\n\n**The vocabulary is a photograph of a corpus.** If medical abbreviations were rare in the training text, they are several tokens each forever after. If a JavaScript idiom was everywhere, it is one token. Two tokenizers trained on different corpora disagree about which words are \"simple\".\n\n**Vocabulary size is a real trade-off.** A bigger vocabulary means shorter sequences (cheaper attention, more text per context window) but the embedding table and the final output layer both scale with it. At a width of 4,096, going from 32,000 to 256,000 vocabulary entries adds roughly 0.9 billion parameters on the input side and the same again on the output side if the two are not shared. That is why small models often carry surprisingly large fractions of their parameters in the vocabulary.\n\n**Multilingual coverage costs vocabulary slots.** Gemma-family models use around 256,000 entries partly to cover many scripts at a reasonable rate. Models with 32,000-entry vocabularies trained mostly on English will shred Devanagari, Tamil or Amharic into near-bytes. That is a design decision with a bill attached, and it lands on the users of those languages.\n\n## The other family: unigram\n\nNot everything is BPE. SentencePiece offers a second method, the unigram language model, which goes the opposite way: start with a large candidate set of substrings, then repeatedly remove the ones whose loss of likelihood is smallest, until you reach the target size. It tends to produce slightly more linguistically sensible pieces and supports sampling different segmentations of the same string, which is used as a regularisation trick during training.\n\nPractically, you will meet both. What matters is that neither one is a parser. A tokenizer never consults grammar or morphology.\n\n## Run one yourself, free\n\n```bash\npip install tokenizers\n```\n\n```python\nfrom tokenizers import Tokenizer, models, trainers, pre_tokenizers\n\ntok = Tokenizer(models.BPE())\ntok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)\ntrainer = trainers.BpeTrainer(vocab_size=2000)\ntok.train([\"your_corpus.txt\"], trainer)\n\nprint(tok.encode(\"thermoregulation\").tokens)\n```\n\nTrain it on ten megabytes of English and look at the splits. Then train an identical run on ten megabytes of Hindi and compare tokens per character. You will produce, in twenty minutes and on a laptop with no GPU, the exact evidence behind the claim that non-English text costs more to process. This is a better use of an afternoon than reading three more articles about tokenisation.\n\n## The thing to remember\n\nThe vocabulary is frozen when the model is trained. You cannot fix a bad split at prompt time, you cannot teach it a new token by asking, and you cannot compare per-token prices across models with different tokenizers without first measuring how many tokens each one produces for your text."},"check":{"question":"Two models both advertise a 128,000-token context window. Your Marathi support transcripts fit comfortably in one but overflow the other. What is the most likely cause?","options":[{"text":"The second model reserves part of its window for internal use, leaving less for input.","correct":false,"tempting":"Some systems really do reserve headroom for the response, but that would not produce a difference of this size between providers."},{"text":"The second model's context limit is measured in characters rather than tokens.","correct":false,"tempting":"Context is sometimes described loosely in articles, so a unit confusion sounds plausible; in practice both are token limits."},{"text":"The transcripts contain formatting that one model strips and the other does not.","correct":false,"tempting":"Providers do differ in preprocessing, which makes this feel like a reasonable operational guess."},{"text":"The two tokenizers were trained on different corpora, so the same Marathi text becomes far more tokens under the one with poorer coverage of the script.","correct":true}]},"prev":{"slug":"tokens-not-words","title":"Tokens, and why a token is not a word","url":"https://addaly.com/learn/how-llms-work/tokens-not-words"},"next":{"slug":"tokenizer-failures-in-the-wild","title":"Tokenizer failures you will actually hit","url":"https://addaly.com/learn/how-llms-work/tokenizer-failures-in-the-wild"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How the vocabulary gets built, by hand', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/building-the-vocabulary (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How the vocabulary gets built, by hand","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"A tokenizer’s vocabulary is the frozen output of a compression run over one corpus, so what it splits cleanly is a fact about that corpus and not about language.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"merge-the-commonest-pair-repeat","runs":[{"text":"Merge the commonest pair, repeat"}]},{"t":"p","runs":[{"text":"The previous lesson said the vocabulary is built by byte pair encoding. Here is the algorithm in full, because it takes five minutes and it removes all the mystery."}]},{"t":"p","runs":[{"text":"Start with a corpus and a vocabulary containing every single byte: 256 entries, which guarantees any text can be represented. Then loop:"}]},{"t":"ol","start":1,"items":[[{"text":"Count every adjacent pair of symbols in the corpus."}],[{"text":"Find the most frequent pair."}],[{"text":"Add it to the vocabulary as a new symbol, and replace every occurrence."}],[{"text":"Stop when the vocabulary reaches the size you wanted."}]]},{"t":"p","runs":[{"text":"Run it on a tiny corpus by hand. Take "},{"text":"low low low lower lowest","code":true},{"text":". Split into characters. The pair "},{"text":"l o","code":true},{"text":" occurs five times, more than any other, so it merges into "},{"text":"lo","code":true},{"text":". Now "},{"text":"lo w","code":true},{"text":" occurs five times, so it merges into "},{"text":"low","code":true},{"text":". Then "},{"text":"low ·","code":true},{"text":" with the following space, and "},{"text":"est","code":true},{"text":", and so on. After a handful of merges, "},{"text":"low","code":true},{"text":" is one symbol and "},{"text":"est","code":true},{"text":" is one symbol, and "},{"text":"lowest","code":true},{"text":" costs two tokens."}]},{"t":"p","runs":[{"text":"Nothing in that procedure knows what a word is. It knows what is frequent."}]},{"t":"code","lang":"python","label":"Python","text":"from collections import Counter\n\ndef merges(words, n):\n    # words: {\"l o w </w>\": 5, \"l o w e r </w>\": 2, ...}\n    for _ in range(n):\n        pairs = Counter()\n        for w, freq in words.items():\n            s = w.split()\n            for a, b in zip(s, s[1:]):\n                pairs[(a, b)] += freq\n        if not pairs:\n            break\n        best = max(pairs, key=pairs.get)\n        joined = \"\".join(best)\n        words = {w.replace(\" \".join(best), joined): f for w, f in words.items()}\n        print(\"merged\", best, \"->\", joined)\n    return words"},{"t":"p","runs":[{"text":"Thirty lines gets you the real thing. The production versions add byte-level fallback, a regular expression that stops merges from crossing certain boundaries, and speed."}]},{"t":"h2","id":"what-that-means-for-you","runs":[{"text":"What that means for you"}]},{"t":"p","runs":[{"text":"Three consequences follow directly, and each one shows up in practice."}]},{"t":"p","runs":[{"text":"The vocabulary is a photograph of a corpus.","b":true},{"text":" If medical abbreviations were rare in the training text, they are several tokens each forever after. If a JavaScript idiom was everywhere, it is one token. Two tokenizers trained on different corpora disagree about which words are “simple”."}]},{"t":"p","runs":[{"text":"Vocabulary size is a real trade-off.","b":true},{"text":" A bigger vocabulary means shorter sequences (cheaper attention, more text per context window) but the embedding table and the final output layer both scale with it. At a width of 4,096, going from 32,000 to 256,000 vocabulary entries adds roughly 0.9 billion parameters on the input side and the same again on the output side if the two are not shared. That is why small models often carry surprisingly large fractions of their parameters in the vocabulary."}]},{"t":"p","runs":[{"text":"Multilingual coverage costs vocabulary slots.","b":true},{"text":" Gemma-family models use around 256,000 entries partly to cover many scripts at a reasonable rate. Models with 32,000-entry vocabularies trained mostly on English will shred Devanagari, Tamil or Amharic into near-bytes. That is a design decision with a bill attached, and it lands on the users of those languages."}]},{"t":"h2","id":"the-other-family-unigram","runs":[{"text":"The other family: unigram"}]},{"t":"p","runs":[{"text":"Not everything is BPE. SentencePiece offers a second method, the unigram language model, which goes the opposite way: start with a large candidate set of substrings, then repeatedly remove the ones whose loss of likelihood is smallest, until you reach the target size. It tends to produce slightly more linguistically sensible pieces and supports sampling different segmentations of the same string, which is used as a regularisation trick during training."}]},{"t":"p","runs":[{"text":"Practically, you will meet both. What matters is that neither one is a parser. A tokenizer never consults grammar or morphology."}]},{"t":"h2","id":"run-one-yourself-free","runs":[{"text":"Run one yourself, free"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install tokenizers"},{"t":"code","lang":"python","label":"Python","text":"from tokenizers import Tokenizer, models, trainers, pre_tokenizers\n\ntok = Tokenizer(models.BPE())\ntok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)\ntrainer = trainers.BpeTrainer(vocab_size=2000)\ntok.train([\"your_corpus.txt\"], trainer)\n\nprint(tok.encode(\"thermoregulation\").tokens)"},{"t":"p","runs":[{"text":"Train it on ten megabytes of English and look at the splits. Then train an identical run on ten megabytes of Hindi and compare tokens per character. You will produce, in twenty minutes and on a laptop with no GPU, the exact evidence behind the claim that non-English text costs more to process. This is a better use of an afternoon than reading three more articles about tokenisation."}]},{"t":"h2","id":"the-thing-to-remember","runs":[{"text":"The thing to remember"}]},{"t":"p","runs":[{"text":"The vocabulary is frozen when the model is trained. You cannot fix a bad split at prompt time, you cannot teach it a new token by asking, and you cannot compare per-token prices across models with different tokenizers without first measuring how many tokens each one produces for your text."}]}],"check":{"question":[{"text":"Two models both advertise a 128,000-token context window. Your Marathi support transcripts fit comfortably in one but overflow the other. What is the most likely cause?"}],"options":[{"text":[{"text":"The second model reserves part of its window for internal use, leaving less for input."}],"correct":false,"tempting":[{"text":"Some systems really do reserve headroom for the response, but that would not produce a difference of this size between providers."}]},{"text":[{"text":"The second model’s context limit is measured in characters rather than tokens."}],"correct":false,"tempting":[{"text":"Context is sometimes described loosely in articles, so a unit confusion sounds plausible; in practice both are token limits."}]},{"text":[{"text":"The transcripts contain formatting that one model strips and the other does not."}],"correct":false,"tempting":[{"text":"Providers do differ in preprocessing, which makes this feel like a reasonable operational guess."}]},{"text":[{"text":"The two tokenizers were trained on different corpora, so the same Marathi text becomes far more tokens under the one with poorer coverage of the script."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"tokens-not-words","title":"Tokens, and why a token is not a word","minutes":8},"next":{"slug":"tokenizer-failures-in-the-wild","title":"Tokenizer failures you will actually hit","minutes":8}}}