Tokens, and why a Hindi bot costs three times an English one

A model does not see characters

Before anything reaches the model, your text is cut into tokens. A tokeniser holds a fixed vocabulary (usually between 32,000 and 200,000 entries) and greedily matches the longest piece it recognises. Common English words are one token each. unbelievable is normally three. A leading space is attached to the word that follows it, which is why hello and hello are different tokens, and why a stray double space quietly costs you money forever.

The working rule for English prose is four characters per token, or about 0.75 tokens per word. A 500-word email is roughly 650 tokens. Memorise that ratio. Every bill, every rate limit and every latency estimate you will make from here on is denominated in tokens, not words.

The rule breaks in three places. One of them will decide whether your product is affordable.

It breaks badly for Indian languages

Almost every widely used tokeniser was fitted to a corpus that is overwhelmingly English. English words earned their own vocabulary entries. Devanagari, Bengali, Tamil and Telugu mostly did not, so they fall back to byte-level pieces: often two or three tokens for a single character, because one Devanagari character is three bytes in UTF-8.

Measure it yourself rather than believing me:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
en = "Hello, how are you? I need help with my order."
hi = "नमस्ते, आप कैसे हैं? मुझे अपने ऑर्डर में मदद चाहिए।"
print(len(enc.encode(en)), len(enc.encode(hi)))

On most current tokenisers the Hindi sentence costs two to four times what the English one costs, for the same meaning. Newer vocabularies have narrowed this (some are down to about 1.5×), but none have closed it. The consequences are concrete:

• A support bot serving Hindi users costs two to four times as much per conversation as the same bot in English.
• The same context window holds two to four times less Hindi.
• Rate limits expressed in tokens per minute bite sooner.

Nobody puts this on a pricing page. It is the single most useful thing in this lesson for anyone building for an Indian, Bangladeshi or Pakistani audience.

The other two breakages are milder. Code is token-hungry because of indentation and punctuation: a well-formatted JSON array of 100 rows spends a large fraction of its tokens re-stating the same keys. And long numbers fragment: 1234567 may be four tokens, which is part of why models are unreliable at arithmetic.

The context window is a budget, not a memory

The advertised window (8k, 128k, 200k, 1M) is the sum of input and output, not the input alone. A model with a 200,000-token window and an 8,192-token output cap cannot write you a 200,000-token document. Two numbers, and they are not the same number.

Budget a real request out loud before you build it:

• System prompt and tool definitions: 400 to 1,500 tokens, on every single call
• Retrieved documents: 4,000 to 8,000 tokens
• Conversation history: grows without limit unless you cap it
• The answer: 300 to 800 tokens

That is a 10,000-token request producing a 500-token reply. At a mid-tier price of roughly $3 per million input tokens and $15 per million output, one call is about 3.75 cents: call it ₹3. Ten thousand of them a day is ₹30,000 a day, and you have not paid for a server yet. Do this arithmetic before you write the feature, not after the invoice.

A big window is not free memory

Two honest limitations, both mechanical.

First, you re-pay for the whole conversation on every turn, because the API is stateless. A chat that reaches 40 turns has resent its early messages 40 times. Cost grows with the square of conversation length, not linearly. Prompt caching (a later lesson) is the fix, and truncation is the fallback.

Second, retrieval accuracy is not flat across the window. Put a needed fact at the start or the end of a long context and models find it reliably. Put the same fact in the middle of 100,000 tokens and measured accuracy drops, sometimes sharply. This has been reproduced across model families for several years and has improved but not disappeared. The practical rule: do not treat a large window as a reason to stop selecting what you send. Ten relevant chunks beat two hundred padded ones, and cost thirty times less.

Count before you send

Every serious tokeniser is free and runs locally: tiktoken for OpenAI-family vocabularies, Hugging Face tokenizers for nearly everything else, llama.cpp ships one for open-weight models. Counting locally costs nothing and takes a millisecond. Refusing a 300,000-token request in your own code is much better than paying for a 400 error, and much, much better than truncating silently and wondering why the answer ignored the last document.