Counting tokens without lying to yourself

The rule of four, and where it breaks

English prose runs at roughly four characters per token. It is a decent envelope estimate and it fails in three places that matter enormously to real systems.

Code. Indentation, brackets and identifiers fragment. Python at four-space indent spends tokens on whitespace; a minified JSON blob spends them on punctuation. Expect 2.5 to 3.5 characters per token for source code: call it 30% worse than prose.

Non-Latin scripts. This is the one that changes the economics of a product. A tokeniser trained mostly on English represents Devanagari, Bengali, Tamil, Thai or Amharic in far more pieces. The same sentence in Hindi commonly costs two to four times the tokens of its English translation; some Indic and African scripts are worse still. Newer tokenisers with larger vocabularies have narrowed the gap without closing it.

The consequences are concrete and unfair: a Hindi-speaking user’s conversation fills the window sooner, costs more per exchange, and is more likely to be truncated. If you serve a multilingual audience, measure the ratio for your actual languages before you set a history budget, and consider setting the budget in tokens per language rather than one number for everybody.

Identifiers and numbers. A UUID is often 15 to 20 tokens. A long table of numeric ids can cost more than the prose describing it.

Count with the right tokeniser

There is no universal token. Every model family has its own vocabulary, and a count from one is only an estimate for another: typically within 10 to 20%, occasionally much worse on non-English text.

Three ways to get a number, in ascending order of trustworthiness:

# 1. Offline, OpenAI-family, exact for those models
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
len(enc.encode(text))

# 2. Offline, exact for any open-weights model, including the chat template
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
len(tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True))

# 3. The provider's own counter: free, and a close estimate of the billed count
client.messages.count_tokens(model=MODEL, system=SYS, messages=msgs, tools=TOOLS)

Both offline options are free, run on a laptop and need no GPU. The third costs a network round trip and is the only one that accounts for a closed vendor’s template and tool serialisation exactly.

The three counts that disagree

You will meet three numbers for the same request, and knowing which is which saves an afternoon:

1. Your estimate, from a local tokeniser. Cheap, approximate for closed models.
2. The provider’s pre-flight count, from a counting endpoint. Includes template and tools; Anthropic documents its count as an estimate that can differ slightly from the billed figure.
3. The usage figure on the response, usage.input_tokens or equivalent. This is the billed number and it is authoritative.

When 1 and 3 differ by more than about 15%, you are missing a block: almost always tool definitions, the chat template, or an image.

Log the billed number on every call. It is the only count you can reason about after the fact, and it is the input to every cost model you will build later.

Budget in tokens, display in characters

A practical asymmetry: your budget arithmetic must be in tokens, but the thing you truncate is text, and truncating text at a token boundary requires the tokeniser. If you are cutting a long document to fit, cut by tokens and decode back:

ids = enc.encode(doc)
clipped = enc.decode(ids[:2000])

Cutting by characters and hoping is how you end up sending half a UTF-8 sequence or slicing a sentence mid-word in a script where that changes the meaning.

Estimating before you have anything to count

You often need a budget before the code exists. A workable envelope, per request:

• System prompt: count it exactly, it is fixed.
• Tool definitions: 150 to 250 tokens per tool, counted once.
• Each retrieved passage: chunk size in tokens plus 20 to 40 for whatever header you attach.
• Each conversation turn: user turns are short and highly variable; assistant turns are roughly your max_tokens at the ceiling and about a third of it in practice.
• Images: 800 to 1,600 each unless you have measured otherwise.

Add 10% for the template and round up. An envelope that is 10% pessimistic is useful; one that is 5% optimistic causes a production incident at the worst possible input.

The limitation to state out loud

You cannot count a closed model’s tokens exactly without asking the vendor, and vendors change tokenisers between model versions. A budget that was 92% full on the old version can overflow on the new one with no change on your side.

This is a real operational risk, not a hypothetical: it is why the overflow lesson later in this module argues for a hard assertion in the assembler rather than a comment saying the request usually fits. Assume your count is wrong by a few percent, and leave the room for it.