Your text is integers, and the count is not the word count

Look at the integers once

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
enc = tok("Tokenisation is unforgiving")
print(enc["input_ids"])
print(tok.convert_ids_to_tokens(enc["input_ids"]))

You get something like ['Token', 'isation', 'Ġis', 'Ġun', 'forg', 'iving']: six tokens for three words. The Ġ is a visible stand-in for a leading space, because byte-level tokenizers treat the space as part of the token that follows it. That is why "cat" and " cat" are different tokens with different ids, and why a stray trailing space in a prompt can change the output.

A tokenizer is a fixed vocabulary (typically 32,000 to 256,000 entries) plus rules for splitting anything into pieces from that vocabulary. Common words get one token. Rare words get chopped. Nothing is ever unrepresentable, because the fallback is bytes.

The ratio, and why it is not the same for everyone

For English prose with an English-centric vocabulary, the working figure is about 1.3 tokens per word, or roughly four characters per token. That number is where the rule of thumb “750 words ≈ 1,000 tokens” comes from.

It does not transfer. The same text in Hindi, Bengali, Tamil, Telugu or Marathi, tokenized by a model whose vocabulary was built mostly from English text, commonly runs two to four times as many tokens for the same meaning, because Devanagari and other Indic scripts fall back to multi-byte pieces. Try it:

for s in ["Where is my parcel?", "मेरा पार्सल कहाँ है?"]:
    print(len(tok(s)["input_ids"]), s)

Three consequences, all practical:

1. Cost. Per-token pricing means the same question costs more in Hindi than in English on the same model. This is a real, measurable inequity in how these systems are priced, and it is a property of the vocabulary rather than of the language.
2. Context. A 4,096-token window holds much less Hindi than English.
3. Quality. More tokens per word means the model has more steps to get right for the same content.

Models with multilingual vocabularies (Qwen, Gemma, Aya, IndicBERT and the muril family) narrow the gap substantially. Checking the ratio on your own text is a ten-second test that should inform model choice and almost never does.

Special tokens appear without being asked for

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tok("hello")["input_ids"])              # [101, 7592, 102]
print(tok.convert_ids_to_tokens([101, 102]))  # ['[CLS]', '[SEP]']

You wrote one word and got three integers. Encoder models wrap input in markers; decoder models often prepend a beginning-of-sequence token. add_special_tokens=False turns this off, which you want when you are assembling a sequence yourself and do not want a second BOS in the middle of it, a common bug when people concatenate a chat template with a manual encode.

Truncation is silent

enc = tok(long_text, truncation=True, max_length=512)

Without truncation=True, a too-long input raises or produces a tensor the model rejects. With it, the text is cut and nothing tells you. A classifier handed a 3,000-word complaint reads the first 512 tokens, which, for a complaint, is the polite introduction rather than the accusation at the end.

If that matters, either chunk deliberately, or pick a model with a longer window, or use truncation_side="left" so you keep the end instead of the beginning.

Offsets: getting back to the characters

enc = tok("Ravi lives in Pune", return_offsets_mapping=True)

Each token comes with the character span it came from. This is how token classification output becomes highlighted text in a UI, and it only works with the fast (Rust) tokenizers, one more reason tokenizer.json in a repository is good news.

Two rules that prevent most tokenizer bugs

• Never mix a tokenizer with a model from a different repository. The integers mean different things in different vocabularies, and nothing checks.
• If you add tokens, resize the embeddings. tok.add_tokens(["<sku>"]) followed by model.resize_token_embeddings(len(tok)). Skip the second line and you get an index error at best, and reads from uninitialised memory at worst.

Do this now

Take one paragraph of your own text (in whatever language you actually work in) and count its tokens under two models: one English-centric like gpt2, one multilingual like Qwen/Qwen2.5-0.5B. Divide by the word count. That ratio is a cost multiplier you will carry through the rest of the course.