{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"the-price-of-a-token","title":"The price of a token, from both sides","url":"https://addaly.com/learn/how-llms-work/the-price-of-a-token","position":65,"of":78,"minutes":9,"answer":"Every line on a price list is a serving mechanism invoiced (output costs more because decode is bandwidth-bound, cached input is cheap because prefill is skipped.","keyIdea":"Every line on a price list is a serving mechanism invoiced (output costs more because decode is bandwidth-bound, cached input is cheap because prefill is skipped, batch is half price because idle capacity is already paid for) and self-hosting beats the API only at high, steady utilisation."},"body":{"format":"markdown","text":"## Reading a price list\n\nEvery API price list has the same shape. Input tokens at one price, output tokens at three to five times that price, a discount for input the provider has seen before, a further discount for work that can wait, and (on reasoning models) a note that the thinking is billed as output whether you see it or not. Each line has a mechanism behind it, and once you have the mechanisms you can predict where the price will move and which of your own levers matters.\n\n## Why output costs more\n\nInput is processed by prefill: every token at once, in a compute-bound pass that uses the card's arithmetic close to fully. Output is produced by decode: one token per step, bandwidth-bound, with the card's arithmetic mostly idle unless the batch is deep. A card can ingest input tokens at a rate many times higher than it can produce output tokens. The price ratio is roughly the ratio of those rates. It is not a value judgement about which tokens are worth more; it is the bandwidth wall, invoiced.\n\n## Why cached input is cheap\n\nModule four described prompt caching: a byte-identical prefix lets the server reuse stored keys and values instead of recomputing them. The provider skips the prefill compute entirely and pays only to keep the blocks in memory for a few minutes. The discount (half to ninety per cent, depending on the provider) reflects that the cost of serving a cached token is storage, not arithmetic. It follows that the discount is only real if your prefix is identical, which is the ordering discipline that lesson insisted on.\n\n## Why batch work is half price\n\nDemand is not flat. A card that is saturated at noon is idle at three in the morning, and idle capacity has already been paid for. A batch endpoint that promises results within a day lets the provider schedule your work into those gaps. Their marginal cost is close to zero, and the discount is how they fill the trough. If your workload can wait (nightly classification, backfills, evaluations) this is the cheapest lever available and it costs you nothing but patience.\n\n## Why the thinking is billed\n\nModule five explained that reasoning models generate long working before the answer. Each token of working is a decode step: the same bandwidth-bound cost as a visible token. Providers may hide the text; they cannot hide the cost. A reasoning model answering a simple question can bill ten times the tokens you can see, which is why routing simple questions away from reasoning models (the lesson after next) is worth real money.\n\n## A worked cost\n\nA support assistant. Each turn sends a 1,500-token system prompt, 2,000 tokens of history, and receives a 300-token reply. Take a mid-tier model priced at $2.50 per million input tokens and $10 per million output.\n\n- Input: 3,500 × $2.50 ÷ 1,000,000 = $0.00875\n- Output: 300 × $10 ÷ 1,000,000 = $0.00300\n- Per turn: about **1.2 cents**\n\nAt ten thousand turns a day, that is about $120 a day and $3,600 a month. Cache the system prompt at a 90 per cent discount and the month falls to around $2,600. Trim the history from 2,000 tokens to 800 with the summarisation that module four described and it falls further, to under $2,000. Move any part of the workload that can wait to the batch endpoint and that part halves again.\n\nThe levers, in the order they usually pay: shorten the context, cache the prefix, batch what can wait, use a smaller model where quality allows, and only then consider running it yourself.\n\n## The self-hosting break-even\n\nRent a data-centre card for around $2 to $3 an hour. From the serving lesson, an 8-billion model on it delivers several thousand output tokens a second at good load: call it ten to fifteen million an hour. That is roughly **$0.20 to $0.30 per million output tokens** at full utilisation. API prices for models of that tier sit in the same range or somewhat above.\n\nNow the honest part. At full utilisation. A card that is busy a fifth of the time costs five times as much per token, which is worse than the API. Self-hosting wins on steady, high, predictable load, on privacy requirements, and on avoiding drift; it loses on bursty or small load, every time. The break-even is a utilisation figure, not a price comparison, and most teams that self-host to save money discover this in the second month.\n\n> Figure: The break-even is a utilisation figure, not a price. Self-hosting wins: Steady, high, predictable load, so the card is busy most of the hour; About $0.20 to $0.30 per million output tokens at full use; A privacy requirement that rules the API out; Drift is intolerable and the weights must be frozen. The API wins: Bursty or small load; A card busy a fifth of the time costs five times as much per token; No capacity to plan, no engine to upgrade, no card to keep warm; Prices have fallen roughly tenfold a year without you doing anything. Rent a card at two to three dollars an hour and the hardware arithmetic and the published price list agree, which is a useful check on both. The levers that pay before this one: shorten the context, cache the prefix, batch what can wait, and use a smaller model where quality allows.\n\nA four-bit 70-billion model on the same card produces a fifth to a tenth as many tokens an hour, so its self-hosted cost is a few dollars per million output tokens, again roughly where the API sits for models of that class. The market is efficient enough that the hardware arithmetic and the price list agree, which is a useful sanity check on both.\n\n## Why prices have fallen so fast\n\nServing got better (continuous batching, paged cache, FP8, speculative decoding) so each card produces more tokens. Models got smaller for the same quality, through distillation, which is the next lesson. Sparse architectures cut the active compute. And there is competition. The result has been a fall of roughly an order of magnitude a year for a fixed level of capability. Design as if it will continue: keep the model swappable, keep the evaluation set ready, and do not sign long contracts at today's prices.\n\n## What nobody outside knows\n\nProviders do not publish their costs, and any confident claim about their margins is an estimate. The arithmetic above is the estimate you can do yourself, with module five's formulas, and it suggests comfortable margins on serving alongside very large training costs that serving must eventually repay. Hold both halves in mind when you read that a price is unsustainable, or that it is a bargain.\n\n## The habit\n\nLog the cost of every request beside its latency, computed from the token counts the response returns. It will move without anyone changing the code: when the prompt grows silently, when a reordering breaks the cache, when a model version changes its verbosity. A cost that is measured per request is a cost that gets noticed the week it changes, rather than at the end of the quarter."},"check":{"question":"On most price lists an output token costs three to five times an input token. What is the mechanism behind the ratio?","options":[{"text":"Output tokens are worth more to the user than input tokens, and providers price according to the value delivered rather than the cost incurred.","correct":false,"tempting":"Value pricing exists in many markets, but here the ratio is remarkably consistent across competing providers, which points to a shared cost structure rather than a shared marketing view."},{"text":"Every output token passes through a safety classifier, which is a second model call that input tokens do not incur.","correct":false,"tempting":"Classifiers do run on outputs at some providers, but they are small models with a tiny fraction of the cost, and the ratio predates their widespread use."},{"text":"Input is processed in one bulk compute-bound pass, while output is produced one bandwidth-bound step at a time.","correct":true},{"text":"Output tokens are sampled, and sampling requires computing a full probability distribution over the vocabulary at every step, which the input tokens skip entirely.","correct":false,"tempting":"The unembedding step is real, but it is a small fraction of a forward pass; the cost difference is in how many tokens each pass produces, not in the final projection."}]},"prev":{"slug":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","url":"https://addaly.com/learn/how-llms-work/splitting-a-model-across-gpus"},"next":{"slug":"distillation-and-small-models","title":"Distillation: how small models got good","url":"https://addaly.com/learn/how-llms-work/distillation-and-small-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The price of a token, from both sides', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/the-price-of-a-token (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The price of a token, from both sides","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"Every line on a price list is a serving mechanism invoiced (output costs more because decode is bandwidth-bound, cached input is cheap because prefill is skipped, batch is half price because idle capacity is already paid for) and self-hosting beats the API only at high, steady utilisation.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Self-hosting wins","items":["Steady, high, predictable load, so the card is busy most of the hour","About $0.20 to $0.30 per million output tokens at full use","A privacy requirement that rules the API out","Drift is intolerable and the weights must be frozen"]},"type":"compare","right":{"title":"The API wins","items":["Bursty or small load","A card busy a fifth of the time costs five times as much per token","No capacity to plan, no engine to upgrade, no card to keep warm","Prices have fallen roughly tenfold a year without you doing anything"]},"title":"The break-even is a utilisation figure, not a price","caption":"Rent a card at two to three dollars an hour and the hardware arithmetic and the published price list agree, which is a useful check on both. The levers that pay before this one: shorten the context, cache the prefix, batch what can wait, and use a smaller model where quality allows."},"alt":"Figure: The break-even is a utilisation figure, not a price. Self-hosting wins: Steady, high, predictable load, so the card is busy most of the hour; About $0.20 to $0.30 per million output tokens at full use; A privacy requirement that rules the API out; Drift is intolerable and the weights must be frozen. The API wins: Bursty or small load; A card busy a fifth of the time costs five times as much per token; No capacity to plan, no engine to upgrade, no card to keep warm; Prices have fallen roughly tenfold a year without you doing anything. Rent a card at two to three dollars an hour and the hardware arithmetic and the published price list agree, which is a useful check on both. The levers that pay before this one: shorten the context, cache the prefix, batch what can wait, and use a smaller model where quality allows."}},"blocks":[{"t":"h2","id":"reading-a-price-list","runs":[{"text":"Reading a price list"}]},{"t":"p","runs":[{"text":"Every API price list has the same shape. Input tokens at one price, output tokens at three to five times that price, a discount for input the provider has seen before, a further discount for work that can wait, and (on reasoning models) a note that the thinking is billed as output whether you see it or not. Each line has a mechanism behind it, and once you have the mechanisms you can predict where the price will move and which of your own levers matters."}]},{"t":"h2","id":"why-output-costs-more","runs":[{"text":"Why output costs more"}]},{"t":"p","runs":[{"text":"Input is processed by prefill: every token at once, in a compute-bound pass that uses the card’s arithmetic close to fully. Output is produced by decode: one token per step, bandwidth-bound, with the card’s arithmetic mostly idle unless the batch is deep. A card can ingest input tokens at a rate many times higher than it can produce output tokens. The price ratio is roughly the ratio of those rates. It is not a value judgement about which tokens are worth more; it is the bandwidth wall, invoiced."}]},{"t":"h2","id":"why-cached-input-is-cheap","runs":[{"text":"Why cached input is cheap"}]},{"t":"p","runs":[{"text":"Module four described prompt caching: a byte-identical prefix lets the server reuse stored keys and values instead of recomputing them. The provider skips the prefill compute entirely and pays only to keep the blocks in memory for a few minutes. The discount (half to ninety per cent, depending on the provider) reflects that the cost of serving a cached token is storage, not arithmetic. It follows that the discount is only real if your prefix is identical, which is the ordering discipline that lesson insisted on."}]},{"t":"h2","id":"why-batch-work-is-half-price","runs":[{"text":"Why batch work is half price"}]},{"t":"p","runs":[{"text":"Demand is not flat. A card that is saturated at noon is idle at three in the morning, and idle capacity has already been paid for. A batch endpoint that promises results within a day lets the provider schedule your work into those gaps. Their marginal cost is close to zero, and the discount is how they fill the trough. If your workload can wait (nightly classification, backfills, evaluations) this is the cheapest lever available and it costs you nothing but patience."}]},{"t":"h2","id":"why-the-thinking-is-billed","runs":[{"text":"Why the thinking is billed"}]},{"t":"p","runs":[{"text":"Module five explained that reasoning models generate long working before the answer. Each token of working is a decode step: the same bandwidth-bound cost as a visible token. Providers may hide the text; they cannot hide the cost. A reasoning model answering a simple question can bill ten times the tokens you can see, which is why routing simple questions away from reasoning models (the lesson after next) is worth real money."}]},{"t":"h2","id":"a-worked-cost","runs":[{"text":"A worked cost"}]},{"t":"p","runs":[{"text":"A support assistant. Each turn sends a 1,500-token system prompt, 2,000 tokens of history, and receives a 300-token reply. Take a mid-tier model priced at $2.50 per million input tokens and $10 per million output."}]},{"t":"ul","items":[[{"text":"Input: 3,500 × $2.50 ÷ 1,000,000 = $0.00875"}],[{"text":"Output: 300 × $10 ÷ 1,000,000 = $0.00300"}],[{"text":"Per turn: about "},{"text":"1.2 cents","b":true}]]},{"t":"p","runs":[{"text":"At ten thousand turns a day, that is about $120 a day and $3,600 a month. Cache the system prompt at a 90 per cent discount and the month falls to around $2,600. Trim the history from 2,000 tokens to 800 with the summarisation that module four described and it falls further, to under $2,000. Move any part of the workload that can wait to the batch endpoint and that part halves again."}]},{"t":"p","runs":[{"text":"The levers, in the order they usually pay: shorten the context, cache the prefix, batch what can wait, use a smaller model where quality allows, and only then consider running it yourself."}]},{"t":"h2","id":"the-self-hosting-break-even","runs":[{"text":"The self-hosting break-even"}]},{"t":"p","runs":[{"text":"Rent a data-centre card for around $2 to $3 an hour. From the serving lesson, an 8-billion model on it delivers several thousand output tokens a second at good load: call it ten to fifteen million an hour. That is roughly "},{"text":"$0.20 to $0.30 per million output tokens","b":true},{"text":" at full utilisation. API prices for models of that tier sit in the same range or somewhat above."}]},{"t":"p","runs":[{"text":"Now the honest part. At full utilisation. A card that is busy a fifth of the time costs five times as much per token, which is worse than the API. Self-hosting wins on steady, high, predictable load, on privacy requirements, and on avoiding drift; it loses on bursty or small load, every time. The break-even is a utilisation figure, not a price comparison, and most teams that self-host to save money discover this in the second month."}]},{"t":"p","runs":[{"text":"A four-bit 70-billion model on the same card produces a fifth to a tenth as many tokens an hour, so its self-hosted cost is a few dollars per million output tokens, again roughly where the API sits for models of that class. The market is efficient enough that the hardware arithmetic and the price list agree, which is a useful sanity check on both."}]},{"t":"h2","id":"why-prices-have-fallen-so-fast","runs":[{"text":"Why prices have fallen so fast"}]},{"t":"p","runs":[{"text":"Serving got better (continuous batching, paged cache, FP8, speculative decoding) so each card produces more tokens. Models got smaller for the same quality, through distillation, which is the next lesson. Sparse architectures cut the active compute. And there is competition. The result has been a fall of roughly an order of magnitude a year for a fixed level of capability. Design as if it will continue: keep the model swappable, keep the evaluation set ready, and do not sign long contracts at today’s prices."}]},{"t":"h2","id":"what-nobody-outside-knows","runs":[{"text":"What nobody outside knows"}]},{"t":"p","runs":[{"text":"Providers do not publish their costs, and any confident claim about their margins is an estimate. The arithmetic above is the estimate you can do yourself, with module five’s formulas, and it suggests comfortable margins on serving alongside very large training costs that serving must eventually repay. Hold both halves in mind when you read that a price is unsustainable, or that it is a bargain."}]},{"t":"h2","id":"the-habit","runs":[{"text":"The habit"}]},{"t":"p","runs":[{"text":"Log the cost of every request beside its latency, computed from the token counts the response returns. It will move without anyone changing the code: when the prompt grows silently, when a reordering breaks the cache, when a model version changes its verbosity. A cost that is measured per request is a cost that gets noticed the week it changes, rather than at the end of the quarter."}]}],"check":{"question":[{"text":"On most price lists an output token costs three to five times an input token. What is the mechanism behind the ratio?"}],"options":[{"text":[{"text":"Output tokens are worth more to the user than input tokens, and providers price according to the value delivered rather than the cost incurred."}],"correct":false,"tempting":[{"text":"Value pricing exists in many markets, but here the ratio is remarkably consistent across competing providers, which points to a shared cost structure rather than a shared marketing view."}]},{"text":[{"text":"Every output token passes through a safety classifier, which is a second model call that input tokens do not incur."}],"correct":false,"tempting":[{"text":"Classifiers do run on outputs at some providers, but they are small models with a tiny fraction of the cost, and the ratio predates their widespread use."}]},{"text":[{"text":"Input is processed in one bulk compute-bound pass, while output is produced one bandwidth-bound step at a time."}],"correct":true,"tempting":null},{"text":[{"text":"Output tokens are sampled, and sampling requires computing a full probability distribution over the vocabulary at every step, which the input tokens skip entirely."}],"correct":false,"tempting":[{"text":"The unembedding step is real, but it is a small fraction of a forward pass; the cost difference is in how many tokens each pass produces, not in the final projection."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","minutes":8},"next":{"slug":"distillation-and-small-models","title":"Distillation: how small models got good","minutes":9}}}