{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"counting-tokens-properly","title":"Counting tokens without lying to yourself","url":"https://addaly.com/learn/context-engineering/counting-tokens-properly","position":6,"of":87,"minutes":9,"answer":"Count with the tokeniser the model actually uses, log the billed count on every response, and remember that non-Latin scripts can cost two to four times more tokens for the same sentence.","keyIdea":"Count with the tokeniser the model actually uses, log the billed count on every response, and remember that non-Latin scripts can cost two to four times more tokens for the same sentence."},"body":{"format":"markdown","text":"## The rule of four, and where it breaks\n\nEnglish prose runs at roughly four characters per token. It is a decent envelope estimate and it fails in three places that matter enormously to real systems.\n\n**Code.** Indentation, brackets and identifiers fragment. Python at four-space indent spends tokens on whitespace; a minified JSON blob spends them on punctuation. Expect 2.5 to 3.5 characters per token for source code: call it 30% worse than prose.\n\n**Non-Latin scripts.** This is the one that changes the economics of a product. A tokeniser trained mostly on English represents Devanagari, Bengali, Tamil, Thai or Amharic in far more pieces. The same sentence in Hindi commonly costs two to four times the tokens of its English translation; some Indic and African scripts are worse still. Newer tokenisers with larger vocabularies have narrowed the gap without closing it.\n\n> Figure: One sentence, five scripts, one tokeniser. Bars (tokens for the same sentence): English 21; Hindi, in Devanagari 49; Bengali 56; Tamil 68; Amharic 77. A tokeniser trained mostly on English cuts other scripts into far more pieces. The consequence is not abstract: the same history budget holds a quarter as much conversation in Tamil as in English, so a Tamil-speaking user is truncated sooner and billed more for the same exchange. Measure the ratio for the languages you actually serve, and consider a budget per language rather than one number for everybody.\n\nThe consequences are concrete and unfair: a Hindi-speaking user's conversation fills the window sooner, costs more per exchange, and is more likely to be truncated. If you serve a multilingual audience, measure the ratio for your actual languages before you set a history budget, and consider setting the budget in tokens per language rather than one number for everybody.\n\n**Identifiers and numbers.** A UUID is often 15 to 20 tokens. A long table of numeric ids can cost more than the prose describing it.\n\n## Count with the right tokeniser\n\nThere is no universal token. Every model family has its own vocabulary, and a count from one is only an estimate for another: typically within 10 to 20%, occasionally much worse on non-English text.\n\nThree ways to get a number, in ascending order of trustworthiness:\n\n```python\n# 1. Offline, OpenAI-family, exact for those models\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nlen(enc.encode(text))\n\n# 2. Offline, exact for any open-weights model, including the chat template\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"meta-llama/Llama-3.1-8B-Instruct\")\nlen(tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True))\n\n# 3. The provider's own counter: free, and a close estimate of the billed count\nclient.messages.count_tokens(model=MODEL, system=SYS, messages=msgs, tools=TOOLS)\n```\n\nBoth offline options are free, run on a laptop and need no GPU. The third costs a network round trip and is the only one that accounts for a closed vendor's template and tool serialisation exactly.\n\n## The three counts that disagree\n\nYou will meet three numbers for the same request, and knowing which is which saves an afternoon:\n\n1. **Your estimate**, from a local tokeniser. Cheap, approximate for closed models.\n2. **The provider's pre-flight count**, from a counting endpoint. Includes template and tools; Anthropic documents its count as an estimate that can differ slightly from the billed figure.\n3. **The usage figure on the response**, `usage.input_tokens` or equivalent. This is the billed number and it is authoritative.\n\nWhen 1 and 3 differ by more than about 15%, you are missing a block: almost always tool definitions, the chat template, or an image.\n\nLog the billed number on every call. It is the only count you can reason about after the fact, and it is the input to every cost model you will build later.\n\n## Budget in tokens, display in characters\n\nA practical asymmetry: your budget arithmetic must be in tokens, but the thing you truncate is text, and truncating text at a token boundary requires the tokeniser. If you are cutting a long document to fit, cut by tokens and decode back:\n\n```python\nids = enc.encode(doc)\nclipped = enc.decode(ids[:2000])\n```\n\nCutting by characters and hoping is how you end up sending half a UTF-8 sequence or slicing a sentence mid-word in a script where that changes the meaning.\n\n## Estimating before you have anything to count\n\nYou often need a budget before the code exists. A workable envelope, per request:\n\n- System prompt: count it exactly, it is fixed.\n- Tool definitions: 150 to 250 tokens per tool, counted once.\n- Each retrieved passage: chunk size in tokens plus 20 to 40 for whatever header you attach.\n- Each conversation turn: user turns are short and highly variable; assistant turns are roughly your `max_tokens` at the ceiling and about a third of it in practice.\n- Images: 800 to 1,600 each unless you have measured otherwise.\n\nAdd 10% for the template and round up. An envelope that is 10% pessimistic is useful; one that is 5% optimistic causes a production incident at the worst possible input.\n\n## The limitation to state out loud\n\nYou cannot count a closed model's tokens exactly without asking the vendor, and vendors change tokenisers between model versions. A budget that was 92% full on the old version can overflow on the new one with no change on your side.\n\nThis is a real operational risk, not a hypothetical: it is why the overflow lesson later in this module argues for a hard assertion in the assembler rather than a comment saying the request usually fits. Assume your count is wrong by a few percent, and leave the room for it."},"check":{"question":"A team sets a 4,000-token history budget using a character-count heuristic tuned on English support chats. Hindi-speaking users report the assistant forgetting things much sooner. What is the mechanism?","options":[{"text":"Hindi text is being truncated mid-character by the byte-level slicing, corrupting the history.","correct":false,"tempting":"Names a real hazard of character slicing but the wrong one; corruption would produce garbled text, not early forgetting."},{"text":"The model has weaker Hindi training, so it fails to use history that is present in the window.","correct":false,"tempting":"Substitutes a capability gap for an accounting gap; the history is genuinely absent, not merely poorly used."},{"text":"The same content in Devanagari costs several times more tokens, so far fewer turns fit inside the same budget.","correct":true},{"text":"Multilingual conversations disable prefix caching, and the uncached prefix is trimmed more aggressively to control cost.","correct":false,"tempting":"Invents a coupling between caching and truncation; caching affects price and latency, not how many turns are retained."}]},"prev":{"slug":"system-prompt-versus-turn","title":"What belongs in the system prompt","url":"https://addaly.com/learn/context-engineering/system-prompt-versus-turn"},"next":{"slug":"the-budget-ledger","title":"Writing the budget down","url":"https://addaly.com/learn/context-engineering/the-budget-ledger"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Counting tokens without lying to yourself', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/counting-tokens-properly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Counting tokens without lying to yourself","courseTitle":"Context Engineering","module":{"index":1,"slug":"the-window-as-a-surface","title":"The window as an engineering surface"},"answer":null,"keyIdea":"Count with the tokeniser the model actually uses, log the billed count on every response, and remember that non-Latin scripts can cost two to four times more tokens for the same sentence.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens for the same sentence","title":"One sentence, five scripts, one tokeniser","series":[{"label":"English","value":21},{"label":"Hindi, in Devanagari","value":49},{"label":"Bengali","value":56},{"label":"Tamil","value":68,"highlight":true},{"label":"Amharic","value":77}],"caption":"A tokeniser trained mostly on English cuts other scripts into far more pieces. The consequence is not abstract: the same history budget holds a quarter as much conversation in Tamil as in English, so a Tamil-speaking user is truncated sooner and billed more for the same exchange. Measure the ratio for the languages you actually serve, and consider a budget per language rather than one number for everybody."},"alt":"Figure: One sentence, five scripts, one tokeniser. Bars (tokens for the same sentence): English 21; Hindi, in Devanagari 49; Bengali 56; Tamil 68; Amharic 77. A tokeniser trained mostly on English cuts other scripts into far more pieces. The consequence is not abstract: the same history budget holds a quarter as much conversation in Tamil as in English, so a Tamil-speaking user is truncated sooner and billed more for the same exchange. Measure the ratio for the languages you actually serve, and consider a budget per language rather than one number for everybody."}},"blocks":[{"t":"h2","id":"the-rule-of-four-and-where-it-breaks","runs":[{"text":"The rule of four, and where it breaks"}]},{"t":"p","runs":[{"text":"English prose runs at roughly four characters per token. It is a decent envelope estimate and it fails in three places that matter enormously to real systems."}]},{"t":"p","runs":[{"text":"Code.","b":true},{"text":" Indentation, brackets and identifiers fragment. Python at four-space indent spends tokens on whitespace; a minified JSON blob spends them on punctuation. Expect 2.5 to 3.5 characters per token for source code: call it 30% worse than prose."}]},{"t":"p","runs":[{"text":"Non-Latin scripts.","b":true},{"text":" This is the one that changes the economics of a product. A tokeniser trained mostly on English represents Devanagari, Bengali, Tamil, Thai or Amharic in far more pieces. The same sentence in Hindi commonly costs two to four times the tokens of its English translation; some Indic and African scripts are worse still. Newer tokenisers with larger vocabularies have narrowed the gap without closing it."}]},{"t":"p","runs":[{"text":"The consequences are concrete and unfair: a Hindi-speaking user’s conversation fills the window sooner, costs more per exchange, and is more likely to be truncated. If you serve a multilingual audience, measure the ratio for your actual languages before you set a history budget, and consider setting the budget in tokens per language rather than one number for everybody."}]},{"t":"p","runs":[{"text":"Identifiers and numbers.","b":true},{"text":" A UUID is often 15 to 20 tokens. A long table of numeric ids can cost more than the prose describing it."}]},{"t":"h2","id":"count-with-the-right-tokeniser","runs":[{"text":"Count with the right tokeniser"}]},{"t":"p","runs":[{"text":"There is no universal token. Every model family has its own vocabulary, and a count from one is only an estimate for another: typically within 10 to 20%, occasionally much worse on non-English text."}]},{"t":"p","runs":[{"text":"Three ways to get a number, in ascending order of trustworthiness:"}]},{"t":"code","lang":"python","label":"Python","text":"# 1. Offline, OpenAI-family, exact for those models\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nlen(enc.encode(text))\n\n# 2. Offline, exact for any open-weights model, including the chat template\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"meta-llama/Llama-3.1-8B-Instruct\")\nlen(tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True))\n\n# 3. The provider's own counter: free, and a close estimate of the billed count\nclient.messages.count_tokens(model=MODEL, system=SYS, messages=msgs, tools=TOOLS)"},{"t":"p","runs":[{"text":"Both offline options are free, run on a laptop and need no GPU. The third costs a network round trip and is the only one that accounts for a closed vendor’s template and tool serialisation exactly."}]},{"t":"h2","id":"the-three-counts-that-disagree","runs":[{"text":"The three counts that disagree"}]},{"t":"p","runs":[{"text":"You will meet three numbers for the same request, and knowing which is which saves an afternoon:"}]},{"t":"ol","start":1,"items":[[{"text":"Your estimate","b":true},{"text":", from a local tokeniser. Cheap, approximate for closed models."}],[{"text":"The provider’s pre-flight count","b":true},{"text":", from a counting endpoint. Includes template and tools; Anthropic documents its count as an estimate that can differ slightly from the billed figure."}],[{"text":"The usage figure on the response","b":true},{"text":", "},{"text":"usage.input_tokens","code":true},{"text":" or equivalent. This is the billed number and it is authoritative."}]]},{"t":"p","runs":[{"text":"When 1 and 3 differ by more than about 15%, you are missing a block: almost always tool definitions, the chat template, or an image."}]},{"t":"p","runs":[{"text":"Log the billed number on every call. It is the only count you can reason about after the fact, and it is the input to every cost model you will build later."}]},{"t":"h2","id":"budget-in-tokens-display-in-characters","runs":[{"text":"Budget in tokens, display in characters"}]},{"t":"p","runs":[{"text":"A practical asymmetry: your budget arithmetic must be in tokens, but the thing you truncate is text, and truncating text at a token boundary requires the tokeniser. If you are cutting a long document to fit, cut by tokens and decode back:"}]},{"t":"code","lang":"python","label":"Python","text":"ids = enc.encode(doc)\nclipped = enc.decode(ids[:2000])"},{"t":"p","runs":[{"text":"Cutting by characters and hoping is how you end up sending half a UTF-8 sequence or slicing a sentence mid-word in a script where that changes the meaning."}]},{"t":"h2","id":"estimating-before-you-have-anything-to-count","runs":[{"text":"Estimating before you have anything to count"}]},{"t":"p","runs":[{"text":"You often need a budget before the code exists. A workable envelope, per request:"}]},{"t":"ul","items":[[{"text":"System prompt: count it exactly, it is fixed."}],[{"text":"Tool definitions: 150 to 250 tokens per tool, counted once."}],[{"text":"Each retrieved passage: chunk size in tokens plus 20 to 40 for whatever header you attach."}],[{"text":"Each conversation turn: user turns are short and highly variable; assistant turns are roughly your "},{"text":"max_tokens","code":true},{"text":" at the ceiling and about a third of it in practice."}],[{"text":"Images: 800 to 1,600 each unless you have measured otherwise."}]]},{"t":"p","runs":[{"text":"Add 10% for the template and round up. An envelope that is 10% pessimistic is useful; one that is 5% optimistic causes a production incident at the worst possible input."}]},{"t":"h2","id":"the-limitation-to-state-out-loud","runs":[{"text":"The limitation to state out loud"}],"role":"limit"},{"t":"p","runs":[{"text":"You cannot count a closed model’s tokens exactly without asking the vendor, and vendors change tokenisers between model versions. A budget that was 92% full on the old version can overflow on the new one with no change on your side."}]},{"t":"p","runs":[{"text":"This is a real operational risk, not a hypothetical: it is why the overflow lesson later in this module argues for a hard assertion in the assembler rather than a comment saying the request usually fits. Assume your count is wrong by a few percent, and leave the room for it."}]}],"check":{"question":[{"text":"A team sets a 4,000-token history budget using a character-count heuristic tuned on English support chats. Hindi-speaking users report the assistant forgetting things much sooner. What is the mechanism?"}],"options":[{"text":[{"text":"Hindi text is being truncated mid-character by the byte-level slicing, corrupting the history."}],"correct":false,"tempting":[{"text":"Names a real hazard of character slicing but the wrong one; corruption would produce garbled text, not early forgetting."}]},{"text":[{"text":"The model has weaker Hindi training, so it fails to use history that is present in the window."}],"correct":false,"tempting":[{"text":"Substitutes a capability gap for an accounting gap; the history is genuinely absent, not merely poorly used."}]},{"text":[{"text":"The same content in Devanagari costs several times more tokens, so far fewer turns fit inside the same budget."}],"correct":true,"tempting":null},{"text":[{"text":"Multilingual conversations disable prefix caching, and the uncached prefix is trimmed more aggressively to control cost."}],"correct":false,"tempting":[{"text":"Invents a coupling between caching and truncation; caching affects price and latency, not how many turns are retained."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"system-prompt-versus-turn","title":"What belongs in the system prompt","minutes":8},"next":{"slug":"the-budget-ledger","title":"Writing the budget down","minutes":8}}}