How the vocabulary gets built, by hand

Merge the commonest pair, repeat

The previous lesson said the vocabulary is built by byte pair encoding. Here is the algorithm in full, because it takes five minutes and it removes all the mystery.

Start with a corpus and a vocabulary containing every single byte: 256 entries, which guarantees any text can be represented. Then loop:

1. Count every adjacent pair of symbols in the corpus.
2. Find the most frequent pair.
3. Add it to the vocabulary as a new symbol, and replace every occurrence.
4. Stop when the vocabulary reaches the size you wanted.

Run it on a tiny corpus by hand. Take low low low lower lowest. Split into characters. The pair l o occurs five times, more than any other, so it merges into lo. Now lo w occurs five times, so it merges into low. Then low · with the following space, and est, and so on. After a handful of merges, low is one symbol and est is one symbol, and lowest costs two tokens.

Nothing in that procedure knows what a word is. It knows what is frequent.

from collections import Counter

def merges(words, n):
    # words: {"l o w </w>": 5, "l o w e r </w>": 2, ...}
    for _ in range(n):
        pairs = Counter()
        for w, freq in words.items():
            s = w.split()
            for a, b in zip(s, s[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        joined = "".join(best)
        words = {w.replace(" ".join(best), joined): f for w, f in words.items()}
        print("merged", best, "->", joined)
    return words

Thirty lines gets you the real thing. The production versions add byte-level fallback, a regular expression that stops merges from crossing certain boundaries, and speed.

What that means for you

Three consequences follow directly, and each one shows up in practice.

The vocabulary is a photograph of a corpus. If medical abbreviations were rare in the training text, they are several tokens each forever after. If a JavaScript idiom was everywhere, it is one token. Two tokenizers trained on different corpora disagree about which words are “simple”.

Vocabulary size is a real trade-off. A bigger vocabulary means shorter sequences (cheaper attention, more text per context window) but the embedding table and the final output layer both scale with it. At a width of 4,096, going from 32,000 to 256,000 vocabulary entries adds roughly 0.9 billion parameters on the input side and the same again on the output side if the two are not shared. That is why small models often carry surprisingly large fractions of their parameters in the vocabulary.

Multilingual coverage costs vocabulary slots. Gemma-family models use around 256,000 entries partly to cover many scripts at a reasonable rate. Models with 32,000-entry vocabularies trained mostly on English will shred Devanagari, Tamil or Amharic into near-bytes. That is a design decision with a bill attached, and it lands on the users of those languages.

The other family: unigram

Not everything is BPE. SentencePiece offers a second method, the unigram language model, which goes the opposite way: start with a large candidate set of substrings, then repeatedly remove the ones whose loss of likelihood is smallest, until you reach the target size. It tends to produce slightly more linguistically sensible pieces and supports sampling different segmentations of the same string, which is used as a regularisation trick during training.

Practically, you will meet both. What matters is that neither one is a parser. A tokenizer never consults grammar or morphology.

Run one yourself, free

pip install tokenizers

from tokenizers import Tokenizer, models, trainers, pre_tokenizers

tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
trainer = trainers.BpeTrainer(vocab_size=2000)
tok.train(["your_corpus.txt"], trainer)

print(tok.encode("thermoregulation").tokens)

Train it on ten megabytes of English and look at the splits. Then train an identical run on ten megabytes of Hindi and compare tokens per character. You will produce, in twenty minutes and on a laptop with no GPU, the exact evidence behind the claim that non-English text costs more to process. This is a better use of an afternoon than reading three more articles about tokenisation.

The thing to remember

The vocabulary is frozen when the model is trained. You cannot fix a bad split at prompt time, you cannot teach it a new token by asking, and you cannot compare per-token prices across models with different tokenizers without first measuring how many tokens each one produces for your text.