Why the same sentence costs different amounts

Tokens are the unit of everything

Price is per token. Context length is in tokens. Speed is tokens per second. So the tokenizer (a lookup table and a merge algorithm, no neural network involved) quietly sets the cost of your product, and it does so differently for every language.

Almost every current model uses byte-level byte-pair encoding or a SentencePiece variant. The training procedure is simple: start from bytes, repeatedly merge the most frequent adjacent pair, stop at the target vocabulary size. Whatever was frequent in the tokenizer’s training text gets a short representation. Whatever was rare gets chopped into pieces.

Vocabulary sizes across the families: Llama 2 used 32,000; Llama 3 moved to 128,256; Qwen uses around 151,000; Gemma uses 256,000. That progression is mostly about multilingual coverage.

The number to measure: fertility

Fertility is tokens per word. Measure it on your own text, because it is the whole story:

from transformers import AutoTokenizer
text_en = "The delivery was delayed by three days and nobody informed me."
text_hi = "डिलीवरी तीन दिन देर से हुई और किसी ने मुझे नहीं बताया।"

for name in ["meta-llama/Llama-2-7b-hf",
             "meta-llama/Llama-3.1-8B",
             "Qwen/Qwen3-8B",
             "google/gemma-3-4b-it"]:
    t = AutoTokenizer.from_pretrained(name)
    print(name, len(t.encode(text_en)), len(t.encode(text_hi)))

The pattern you will see: English costs roughly 1.2 to 1.4 tokens per word everywhere. Hindi in Devanagari on an older 32k tokenizer can cost five to eight tokens per word, because each Devanagari character is three bytes in UTF-8 and, with few merges learned for those byte sequences, the tokenizer falls back to something close to byte-per-token. A newer large-vocabulary tokenizer typically brings that down to two or three.

Three consequences follow directly, and none of them is about quality of writing:

• Cost. At three times the fertility you pay three times as much per sentence, on identical hardware and an identical price list.
• Context. A 32,000-token window holds perhaps 24,000 English words, or 6,000 Hindi words on a bad tokenizer. The same document does not fit.
• Speed. Generation is per token, so a reply of the same length takes three times as long.

This is the mechanical part of why a language-tuned model is often the right choice: the tokenizer usually comes with it.

Numbers, code and the details that surprise

Digits. Llama 3 splits numbers into groups of up to three digits; several other families split every digit separately. This is deliberate (consistent digit tokenisation makes arithmetic more learnable) and it means a long number is many tokens. It also means that arithmetic failures are partly a representation problem, not only a reasoning one.

Leading spaces. In byte-level BPE, hello and hello are different tokens. This is behind a whole family of odd bugs when you build prompts by string concatenation, and behind the trick called token healing that some libraries apply.

Code. Indentation matters: tokenizers trained with code merge runs of spaces, so four spaces can be one token or four depending on the family. On a large codebase this is a real cost difference.

Whitespace and newlines are tokens too, which is why heavily formatted prompts with many blank lines cost more than they look.

The tokenizer is part of the model

You cannot mix them. Loading Qwen weights with a Llama tokenizer produces text that is grammatical nonsense, because token 5000 means something different in each. When you download a model, take tokenizer.json and tokenizer_config.json from the same revision as the weights.

This also constrains fine-tuning. You can add tokens to a vocabulary, but every added token needs a new row in the embedding matrix and in the output head, initialised somehow, and the model has never seen those rows. Adding vocabulary for a new language is a real technique and it requires continued pretraining to be worth anything. A later lesson covers it.

A five-minute exercise worth doing once

Take a thousand real documents from your own system. Run them through the tokenizers of the three models you are considering. Compute the mean tokens per document for each.

That number, multiplied by your request volume and the provider’s price, is your bill. Teams routinely discover a 30% cost difference between two models of the same size that they were choosing between on quality alone.