The arithmetic of a request

Four prices, not one

Every hosted model has at least four rates, and confusing them is how a cost model ends up wrong by a factor of five.

• Input, per million tokens. The base rate.
• Output, per million tokens. Typically four to eight times the input rate (five times on every current Anthropic model and on OpenAI’s newest; six to eight times on several OpenAI and Google models, checked October 2026), because generation is sequential and cannot be batched the same way.
• Cached input, when a prefix is reused. Often around a tenth of the base input rate.
• Cache write, a premium on the first call that stores the prefix: commonly around 1.25 times base input.

There is usually a batch rate as well, around half price for work that can wait hours.

Exact numbers change every few months and differ by an order of magnitude between the cheapest small model and the largest frontier one, so look them up rather than memorising them. The ratios above are stable and are what the reasoning depends on.

The formula for one call:

cost = uncached_in × p_in
     + cached_in   × p_in × 0.1
     + written_in  × p_in × 1.25
     + out         × p_out

Every response tells you those four counts in its usage block. Log them. A cost model built from logged usage is right; one built from estimates drifts within a month.

The number that surprises everybody

A conversation is billed quadratically.

Each turn resends the entire history. Ten turns of 500 tokens each are not 5,000 tokens of billing. They are 500 + 1,000 + 1,500 + ... + 5,000 = 27,500 input tokens, for 5,000 tokens of actual content. Twenty turns is 105,000. The content grew linearly; the bill grew with the square.

This is why a chat product’s cost per user is dominated by long conversations rather than by many short ones, and why the average conversation length is one of the most important numbers in your cost model. It is also the strongest argument for prefix caching, which turns most of that resent history into cached reads at a tenth of the price, and for the compaction techniques in the next module.

Do the arithmetic for your own product before you price it. A team assuming linear growth will underestimate a twenty-turn conversation by roughly a factor of ten.

Where the money actually is

Run this for one real request and you will usually find the same shape:

• Retrieved passages: the largest single block, often half the input.
• History: second, and growing.
• Tool definitions: a fixed tax, larger than teams expect.
• System prompt and exemplars: smaller than everyone assumes.
• Output: few tokens, high rate. At 3% of the tokens and a 5x price it is about 13% of an uncached bill (about 20% at 8x), and it climbs toward a third or more when most of the input is billed at cache-read rates.

Two consequences for where to spend an afternoon. Cutting output length is disproportionately valuable, because output is the expensive rate; asking for a shorter answer is a genuine cost lever, not a stylistic one. And cutting retrieved passages from ten to five, which the k-curve lesson may already have told you to do for quality reasons, is usually the largest single saving available.

Cost per unit of value, not cost per call

Cost per call is the wrong denominator for a decision. The right one is cost per resolved ticket, per correct extraction, per accepted draft.

A change that halves the cost per call and raises the failure rate from 8% to 20% has increased the cost per resolution, because every failure is retried, escalated to a human, or abandoned. A human escalation typically costs orders of magnitude more than any inference call, so a small accuracy loss can swamp a large token saving.

Write it down explicitly:

cost_per_resolution = (calls × cost_per_call) / resolved

This is the number that survives contact with a finance conversation, and it is the one that prevents the classic mistake of celebrating a 40% infrastructure saving that quietly doubled the support workload.

Estimating before you build

A usable envelope, per feature:

1. Requests per day.
2. Input tokens per request, from the budget ledger in module one.
3. Output tokens per request, at the 95th percentile rather than the mean.
4. Expected cache hit rate: assume 0 until measured.

Multiply, then double it. The doubling is not pessimism; it covers retries, evaluation runs, the internal team’s usage, and the requests that turn out to be four calls rather than one because a tool was invoked.

A forecast that comes in at twice the eventual bill costs you a conversation. One that comes in at half costs you the project’s credibility in month two.