{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"tokens-and-the-context-window","title":"Tokens, and why a Hindi bot costs three times an English one","url":"https://addaly.com/learn/building-with-ai/tokens-and-the-context-window","position":3,"of":80,"minutes":10,"answer":"Cost, limits and latency are all measured in tokens, and the same sentence in Hindi can cost three times what it costs in English because the vocabulary was fitted to English.","keyIdea":"Cost, limits and latency are all measured in tokens, and the same sentence in Hindi can cost three times what it costs in English because the vocabulary was fitted to English."},"body":{"format":"markdown","text":"## A model does not see characters\n\nBefore anything reaches the model, your text is cut into **tokens**. A tokeniser holds a fixed vocabulary (usually between 32,000 and 200,000 entries) and greedily matches the longest piece it recognises. Common English words are one token each. `unbelievable` is normally three. A leading space is attached to the word that follows it, which is why `hello` and ` hello` are different tokens, and why a stray double space quietly costs you money forever.\n\nThe working rule for English prose is **four characters per token**, or about 0.75 tokens per word. A 500-word email is roughly 650 tokens. Memorise that ratio. Every bill, every rate limit and every latency estimate you will make from here on is denominated in tokens, not words.\n\nThe rule breaks in three places. One of them will decide whether your product is affordable.\n\n### It breaks badly for Indian languages\n\nAlmost every widely used tokeniser was fitted to a corpus that is overwhelmingly English. English words earned their own vocabulary entries. Devanagari, Bengali, Tamil and Telugu mostly did not, so they fall back to byte-level pieces: often two or three tokens for a single character, because one Devanagari character is three bytes in UTF-8.\n\nMeasure it yourself rather than believing me:\n\n```python\nimport tiktoken\nenc = tiktoken.get_encoding(\"cl100k_base\")\nen = \"Hello, how are you? I need help with my order.\"\nhi = \"नमस्ते, आप कैसे हैं? मुझे अपने ऑर्डर में मदद चाहिए।\"\nprint(len(enc.encode(en)), len(enc.encode(hi)))\n```\n\nOn most current tokenisers the Hindi sentence costs two to four times what the English one costs, for the same meaning. Newer vocabularies have narrowed this (some are down to about 1.5×), but none have closed it. The consequences are concrete:\n\n- A support bot serving Hindi users costs two to four times as much per conversation as the same bot in English.\n- The same context window holds two to four times less Hindi.\n- Rate limits expressed in tokens per minute bite sooner.\n\nNobody puts this on a pricing page. It is the single most useful thing in this lesson for anyone building for an Indian, Bangladeshi or Pakistani audience.\n\nThe other two breakages are milder. Code is token-hungry because of indentation and punctuation: a well-formatted JSON array of 100 rows spends a large fraction of its tokens re-stating the same keys. And long numbers fragment: `1234567` may be four tokens, which is part of why models are unreliable at arithmetic.\n\n## The context window is a budget, not a memory\n\nThe advertised window (8k, 128k, 200k, 1M) is the **sum of input and output**, not the input alone. A model with a 200,000-token window and an 8,192-token output cap cannot write you a 200,000-token document. Two numbers, and they are not the same number.\n\nBudget a real request out loud before you build it:\n\n> Figure: One sentence, four scripts, four bills. Bars (tokens for the same sentence): English 12; Hindi in Roman letters 17; Hindi in Devanagari 38; Tamil in Tamil script 46. \"Hello, how are you? I need help with my order.\" and its translations, through a common English-fitted tokeniser. Devanagari and Tamil fall back to byte-level pieces, so a support bot in Hindi costs about three times the same bot in English and its context window holds three times less.\n\n- System prompt and tool definitions: 400 to 1,500 tokens, on every single call\n- Retrieved documents: 4,000 to 8,000 tokens\n- Conversation history: grows without limit unless you cap it\n- The answer: 300 to 800 tokens\n\nThat is a 10,000-token request producing a 500-token reply. At a mid-tier price of roughly $3 per million input tokens and $15 per million output, one call is about 3.75 cents: call it ₹3. Ten thousand of them a day is ₹30,000 a day, and you have not paid for a server yet. Do this arithmetic before you write the feature, not after the invoice.\n\n## A big window is not free memory\n\nTwo honest limitations, both mechanical.\n\nFirst, **you re-pay for the whole conversation on every turn**, because the API is stateless. A chat that reaches 40 turns has resent its early messages 40 times. Cost grows with the square of conversation length, not linearly. Prompt caching (a later lesson) is the fix, and truncation is the fallback.\n\nSecond, **retrieval accuracy is not flat across the window**. Put a needed fact at the start or the end of a long context and models find it reliably. Put the same fact in the middle of 100,000 tokens and measured accuracy drops, sometimes sharply. This has been reproduced across model families for several years and has improved but not disappeared. The practical rule: do not treat a large window as a reason to stop selecting what you send. Ten relevant chunks beat two hundred padded ones, and cost thirty times less.\n\n## Count before you send\n\nEvery serious tokeniser is free and runs locally: `tiktoken` for OpenAI-family vocabularies, Hugging Face `tokenizers` for nearly everything else, `llama.cpp` ships one for open-weight models. Counting locally costs nothing and takes a millisecond. Refusing a 300,000-token request in your own code is much better than paying for a 400 error, and much, much better than truncating silently and wondering why the answer ignored the last document."},"check":{"question":"A support bot handles the same number of conversations in Hindi as in English, with the same prompts, and the Hindi bill is three times larger. What is happening?","options":[{"text":"The provider charges a higher per-token rate for non-English text.","correct":false,"tempting":"Assumes pricing is language-aware; the rate is per token and identical, which is exactly why the token count is where the difference hides."},{"text":"The tokeniser's vocabulary was learned mostly from English, so Devanagari falls back to byte-level pieces and the same meaning costs more tokens.","correct":true},{"text":"Hindi speakers write longer messages, so more text is being sent.","correct":false,"tempting":"Blames user behaviour for a difference that persists when you encode two exactly equivalent sentences and compare counts."},{"text":"The model translates internally to English and back, and you pay for both passes.","correct":false,"tempting":"Imagines a hidden translation step; the model processes the tokens it is given in one pass and nothing is translated behind the scenes."}]},"prev":{"slug":"system-prompts-and-turns","title":"System prompts, user turns, and who the model listens to","url":"https://addaly.com/learn/building-with-ai/system-prompts-and-turns"},"next":{"slug":"sampling-and-determinism","title":"Temperature, sampling, and why the same input gives a different answer","url":"https://addaly.com/learn/building-with-ai/sampling-and-determinism"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Tokens, and why a Hindi bot costs three times an English one', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/tokens-and-the-context-window (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Tokens, and why a Hindi bot costs three times an English one","courseTitle":"Building with AI","module":{"index":1,"slug":"the-call-underneath-everything","title":"The call underneath everything"},"answer":null,"keyIdea":"Cost, limits and latency are all measured in tokens, and the same sentence in Hindi can cost three times what it costs in English because the vocabulary was fitted to English.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens for the same sentence","title":"One sentence, four scripts, four bills","series":[{"label":"English","value":12},{"label":"Hindi in Roman letters","value":17},{"label":"Hindi in Devanagari","value":38,"highlight":true},{"label":"Tamil in Tamil script","value":46}],"caption":"“Hello, how are you? I need help with my order.” and its translations, through a common English-fitted tokeniser. Devanagari and Tamil fall back to byte-level pieces, so a support bot in Hindi costs about three times the same bot in English and its context window holds three times less."},"alt":"Figure: One sentence, four scripts, four bills. Bars (tokens for the same sentence): English 12; Hindi in Roman letters 17; Hindi in Devanagari 38; Tamil in Tamil script 46. “Hello, how are you? I need help with my order.” and its translations, through a common English-fitted tokeniser. Devanagari and Tamil fall back to byte-level pieces, so a support bot in Hindi costs about three times the same bot in English and its context window holds three times less."}},"blocks":[{"t":"h2","id":"a-model-does-not-see-characters","runs":[{"text":"A model does not see characters"}]},{"t":"p","runs":[{"text":"Before anything reaches the model, your text is cut into "},{"text":"tokens","b":true},{"text":". A tokeniser holds a fixed vocabulary (usually between 32,000 and 200,000 entries) and greedily matches the longest piece it recognises. Common English words are one token each. "},{"text":"unbelievable","code":true},{"text":" is normally three. A leading space is attached to the word that follows it, which is why "},{"text":"hello","code":true},{"text":" and "},{"text":" hello","code":true},{"text":" are different tokens, and why a stray double space quietly costs you money forever."}]},{"t":"p","runs":[{"text":"The working rule for English prose is "},{"text":"four characters per token","b":true},{"text":", or about 0.75 tokens per word. A 500-word email is roughly 650 tokens. Memorise that ratio. Every bill, every rate limit and every latency estimate you will make from here on is denominated in tokens, not words."}]},{"t":"p","runs":[{"text":"The rule breaks in three places. One of them will decide whether your product is affordable."}]},{"t":"h3","id":"it-breaks-badly-for-indian-languages","runs":[{"text":"It breaks badly for Indian languages"}]},{"t":"p","runs":[{"text":"Almost every widely used tokeniser was fitted to a corpus that is overwhelmingly English. English words earned their own vocabulary entries. Devanagari, Bengali, Tamil and Telugu mostly did not, so they fall back to byte-level pieces: often two or three tokens for a single character, because one Devanagari character is three bytes in UTF-8."}]},{"t":"p","runs":[{"text":"Measure it yourself rather than believing me:"}]},{"t":"code","lang":"python","label":"Python","text":"import tiktoken\nenc = tiktoken.get_encoding(\"cl100k_base\")\nen = \"Hello, how are you? I need help with my order.\"\nhi = \"नमस्ते, आप कैसे हैं? मुझे अपने ऑर्डर में मदद चाहिए।\"\nprint(len(enc.encode(en)), len(enc.encode(hi)))"},{"t":"p","runs":[{"text":"On most current tokenisers the Hindi sentence costs two to four times what the English one costs, for the same meaning. Newer vocabularies have narrowed this (some are down to about 1.5×), but none have closed it. The consequences are concrete:"}]},{"t":"ul","items":[[{"text":"A support bot serving Hindi users costs two to four times as much per conversation as the same bot in English."}],[{"text":"The same context window holds two to four times less Hindi."}],[{"text":"Rate limits expressed in tokens per minute bite sooner."}]]},{"t":"p","runs":[{"text":"Nobody puts this on a pricing page. It is the single most useful thing in this lesson for anyone building for an Indian, Bangladeshi or Pakistani audience."}]},{"t":"p","runs":[{"text":"The other two breakages are milder. Code is token-hungry because of indentation and punctuation: a well-formatted JSON array of 100 rows spends a large fraction of its tokens re-stating the same keys. And long numbers fragment: "},{"text":"1234567","code":true},{"text":" may be four tokens, which is part of why models are unreliable at arithmetic."}]},{"t":"h2","id":"the-context-window-is-a-budget-not-a-memory","runs":[{"text":"The context window is a budget, not a memory"}]},{"t":"p","runs":[{"text":"The advertised window (8k, 128k, 200k, 1M) is the "},{"text":"sum of input and output","b":true},{"text":", not the input alone. A model with a 200,000-token window and an 8,192-token output cap cannot write you a 200,000-token document. Two numbers, and they are not the same number."}]},{"t":"p","runs":[{"text":"Budget a real request out loud before you build it:"}]},{"t":"ul","items":[[{"text":"System prompt and tool definitions: 400 to 1,500 tokens, on every single call"}],[{"text":"Retrieved documents: 4,000 to 8,000 tokens"}],[{"text":"Conversation history: grows without limit unless you cap it"}],[{"text":"The answer: 300 to 800 tokens"}]]},{"t":"p","runs":[{"text":"That is a 10,000-token request producing a 500-token reply. At a mid-tier price of roughly $3 per million input tokens and $15 per million output, one call is about 3.75 cents: call it ₹3. Ten thousand of them a day is ₹30,000 a day, and you have not paid for a server yet. Do this arithmetic before you write the feature, not after the invoice."}]},{"t":"h2","id":"a-big-window-is-not-free-memory","runs":[{"text":"A big window is not free memory"}]},{"t":"p","runs":[{"text":"Two honest limitations, both mechanical."}]},{"t":"p","runs":[{"text":"First, "},{"text":"you re-pay for the whole conversation on every turn","b":true},{"text":", because the API is stateless. A chat that reaches 40 turns has resent its early messages 40 times. Cost grows with the square of conversation length, not linearly. Prompt caching (a later lesson) is the fix, and truncation is the fallback."}]},{"t":"p","runs":[{"text":"Second, "},{"text":"retrieval accuracy is not flat across the window","b":true},{"text":". Put a needed fact at the start or the end of a long context and models find it reliably. Put the same fact in the middle of 100,000 tokens and measured accuracy drops, sometimes sharply. This has been reproduced across model families for several years and has improved but not disappeared. The practical rule: do not treat a large window as a reason to stop selecting what you send. Ten relevant chunks beat two hundred padded ones, and cost thirty times less."}]},{"t":"h2","id":"count-before-you-send","runs":[{"text":"Count before you send"}]},{"t":"p","runs":[{"text":"Every serious tokeniser is free and runs locally: "},{"text":"tiktoken","code":true},{"text":" for OpenAI-family vocabularies, Hugging Face "},{"text":"tokenizers","code":true},{"text":" for nearly everything else, "},{"text":"llama.cpp","code":true},{"text":" ships one for open-weight models. Counting locally costs nothing and takes a millisecond. Refusing a 300,000-token request in your own code is much better than paying for a 400 error, and much, much better than truncating silently and wondering why the answer ignored the last document."}]}],"check":{"question":[{"text":"A support bot handles the same number of conversations in Hindi as in English, with the same prompts, and the Hindi bill is three times larger. What is happening?"}],"options":[{"text":[{"text":"The provider charges a higher per-token rate for non-English text."}],"correct":false,"tempting":[{"text":"Assumes pricing is language-aware; the rate is per token and identical, which is exactly why the token count is where the difference hides."}]},{"text":[{"text":"The tokeniser’s vocabulary was learned mostly from English, so Devanagari falls back to byte-level pieces and the same meaning costs more tokens."}],"correct":true,"tempting":null},{"text":[{"text":"Hindi speakers write longer messages, so more text is being sent."}],"correct":false,"tempting":[{"text":"Blames user behaviour for a difference that persists when you encode two exactly equivalent sentences and compare counts."}]},{"text":[{"text":"The model translates internally to English and back, and you pay for both passes."}],"correct":false,"tempting":[{"text":"Imagines a hidden translation step; the model processes the tokens it is given in one pass and nothing is translated behind the scenes."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"system-prompts-and-turns","title":"System prompts, user turns, and who the model listens to","minutes":6},"next":{"slug":"sampling-and-determinism","title":"Temperature, sampling, and why the same input gives a different answer","minutes":9}}}