The price of a token, from both sides

Reading a price list

Every API price list has the same shape. Input tokens at one price, output tokens at three to five times that price, a discount for input the provider has seen before, a further discount for work that can wait, and (on reasoning models) a note that the thinking is billed as output whether you see it or not. Each line has a mechanism behind it, and once you have the mechanisms you can predict where the price will move and which of your own levers matters.

Why output costs more

Input is processed by prefill: every token at once, in a compute-bound pass that uses the card’s arithmetic close to fully. Output is produced by decode: one token per step, bandwidth-bound, with the card’s arithmetic mostly idle unless the batch is deep. A card can ingest input tokens at a rate many times higher than it can produce output tokens. The price ratio is roughly the ratio of those rates. It is not a value judgement about which tokens are worth more; it is the bandwidth wall, invoiced.

Why cached input is cheap

Module four described prompt caching: a byte-identical prefix lets the server reuse stored keys and values instead of recomputing them. The provider skips the prefill compute entirely and pays only to keep the blocks in memory for a few minutes. The discount (half to ninety per cent, depending on the provider) reflects that the cost of serving a cached token is storage, not arithmetic. It follows that the discount is only real if your prefix is identical, which is the ordering discipline that lesson insisted on.

Why batch work is half price

Demand is not flat. A card that is saturated at noon is idle at three in the morning, and idle capacity has already been paid for. A batch endpoint that promises results within a day lets the provider schedule your work into those gaps. Their marginal cost is close to zero, and the discount is how they fill the trough. If your workload can wait (nightly classification, backfills, evaluations) this is the cheapest lever available and it costs you nothing but patience.

Why the thinking is billed

Module five explained that reasoning models generate long working before the answer. Each token of working is a decode step: the same bandwidth-bound cost as a visible token. Providers may hide the text; they cannot hide the cost. A reasoning model answering a simple question can bill ten times the tokens you can see, which is why routing simple questions away from reasoning models (the lesson after next) is worth real money.

A worked cost

A support assistant. Each turn sends a 1,500-token system prompt, 2,000 tokens of history, and receives a 300-token reply. Take a mid-tier model priced at $2.50 per million input tokens and $10 per million output.

• Input: 3,500 × $2.50 ÷ 1,000,000 = $0.00875
• Output: 300 × $10 ÷ 1,000,000 = $0.00300
• Per turn: about 1.2 cents

At ten thousand turns a day, that is about $120 a day and $3,600 a month. Cache the system prompt at a 90 per cent discount and the month falls to around $2,600. Trim the history from 2,000 tokens to 800 with the summarisation that module four described and it falls further, to under $2,000. Move any part of the workload that can wait to the batch endpoint and that part halves again.

The levers, in the order they usually pay: shorten the context, cache the prefix, batch what can wait, use a smaller model where quality allows, and only then consider running it yourself.

The self-hosting break-even

Rent a data-centre card for around $2 to $3 an hour. From the serving lesson, an 8-billion model on it delivers several thousand output tokens a second at good load: call it ten to fifteen million an hour. That is roughly $0.20 to $0.30 per million output tokens at full utilisation. API prices for models of that tier sit in the same range or somewhat above.

Now the honest part. At full utilisation. A card that is busy a fifth of the time costs five times as much per token, which is worse than the API. Self-hosting wins on steady, high, predictable load, on privacy requirements, and on avoiding drift; it loses on bursty or small load, every time. The break-even is a utilisation figure, not a price comparison, and most teams that self-host to save money discover this in the second month.

A four-bit 70-billion model on the same card produces a fifth to a tenth as many tokens an hour, so its self-hosted cost is a few dollars per million output tokens, again roughly where the API sits for models of that class. The market is efficient enough that the hardware arithmetic and the price list agree, which is a useful sanity check on both.

Why prices have fallen so fast

Serving got better (continuous batching, paged cache, FP8, speculative decoding) so each card produces more tokens. Models got smaller for the same quality, through distillation, which is the next lesson. Sparse architectures cut the active compute. And there is competition. The result has been a fall of roughly an order of magnitude a year for a fixed level of capability. Design as if it will continue: keep the model swappable, keep the evaluation set ready, and do not sign long contracts at today’s prices.

What nobody outside knows

Providers do not publish their costs, and any confident claim about their margins is an estimate. The arithmetic above is the estimate you can do yourself, with module five’s formulas, and it suggests comfortable margins on serving alongside very large training costs that serving must eventually repay. Hold both halves in mind when you read that a price is unsustainable, or that it is a bargain.

The habit

Log the cost of every request beside its latency, computed from the token counts the response returns. It will move without anyone changing the code: when the prompt grows silently, when a reordering breaks the cache, when a model version changes its verbosity. A cost that is measured per request is a cost that gets noticed the week it changes, rather than at the end of the quarter.