{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"cost-per-task","title":"Cost per task, not cost per token","url":"https://addaly.com/learn/open-models/cost-per-task","position":69,"of":71,"minutes":8,"answer":"Cost is tokens per task times price times retries.","keyIdea":"Cost is tokens per task times price times retries, so a cheaper model carrying a long few-shot prompt or a reasoning model spending output tokens on thinking can cost several times more than a dearer model with a short prompt."},"body":{"format":"markdown","text":"## The price list is not the cost\n\nA model at 15 cents per million tokens is not cheaper than one at 60 cents until you know how many tokens each spends on your task. Three things routinely invert the ranking.\n\n**Tokens per task.** A weaker model that needs eight few-shot examples in every prompt carries those examples on every request. A stronger model that works with three sentences of instruction does not. Two thousand tokens of examples at a hundred thousand requests a month is 200 million input tokens that the cheaper model spends and the dearer one does not.\n\n**Retries.** A model that produces invalid output 6% of the time costs 1.06 attempts per task, plus the engineering and the latency. At 20% it costs 1.25 attempts and a support problem.\n\n**Thinking tokens.** A reasoning model's chain of thought is billed as output, and output is priced two to four times input. Three to twenty times the output tokens for the same visible answer is the range from the earlier lesson, and it can make a model with an attractive price the most expensive option available.\n\n## The formula\n\n```\ncost per task = (input_tokens  x input_price\n               + output_tokens x output_price)\n              x (1 + retry_rate)\n```\n\nEverything else is a detail of how you count the first four terms.\n\n## A worked comparison\n\nA support classification task, 100,000 requests a month. Measured, not guessed:\n\n**Rented 8B, few-shot prompt.** 1,400 input tokens (900 of examples), 60 output. At 20 cents per million in and 40 out: 1,400 × 0.20 + 60 × 0.40 = 304 microdollars per task, about **30 dollars a month**. Retry rate 4%, so about 32.\n\n**Rented 70B, short prompt.** 400 input, 60 output. At 2.00 in and 2.40 out: 400 × 2.00 + 60 × 2.40 = 944 microdollars, about **94 dollars a month**. Retry rate 1%.\n\n**Reasoning model, short prompt.** 400 input, 900 output including thinking. Even at the 8B's prices that is 1,240 microdollars (**124 dollars a month**) from a model whose price list looked cheap.\n\n**Self-hosted tuned 3B.** No prompt examples needed: 200 input, 60 output. On an always-free CPU instance for a batch workload, or a few dollars a month of a shared GPU. Plus the fine-tune's development cost and its permanent maintenance.\n\n> Figure: One classification task, a hundred thousand requests a month. Bars (dollars a month, measured rather than estimated): Reasoning model, 900 output tokens of thinking 124; Rented 70B, 400-token prompt 94; Rented 8B, 900 tokens of examples 32; Self-hosted tuned 3B, no examples 4. The ranking is not the ranking of the price list. The reasoning model's per-million price looked cheap and it spends nine hundred output tokens thinking, billed at two to four times the input rate. The tuned 3B is cheapest to run and carries a fine-tune's development cost and permanent maintenance.\n\nThe ranking here is not the ranking of the price list, and it moves again if the volume is ten times larger or ten times smaller.\n\n## Prefix caching changes the arithmetic\n\nIf every request starts with the same 900 tokens of examples and system prompt, prefix caching means that portion is computed once and reused. Where a provider offers cached-input pricing it is typically a fraction of the standard input price; where you self-host, the saving is compute rather than money but shows up as capacity.\n\nThis is the single change that most often makes a few-shot prompt affordable, and it is a configuration flag rather than a project. Structure your prompt so the constant part comes first and the variable part last, or caching cannot help you.\n\n## The self-hosted version\n\n```\ncost per task = (GPU cost per hour) / (tasks completed per hour)\n```\n\nThe denominator is where batching lives, and it is why the utilisation argument from the rent-or-own lesson matters here too. The same GPU serving one request at a time and serving thirty concurrently has the same numerator and a wildly different denominator.\n\n## What to actually do\n\nInstrument it. Log input tokens, output tokens, retries and the model for every request. Then cost per task is a query rather than an estimate, and you can see it change when somebody adds two paragraphs to the prompt.\n\n```json\n{\"task\":\"triage\",\"model\":\"qwen3-8b\",\"prompt_tok\":1412,\n \"completion_tok\":58,\"attempt\":1,\"ms\":840}\n```\n\nSix fields. Without them, every cost conversation is a guess, and the first thing anybody does in an optimisation exercise is spend a week collecting exactly this."},"check":{"question":"Team A uses a model at 20 cents per million input tokens with a 1,400-token prompt. Team B uses one at 2 dollars per million with a 400-token prompt. Output is 60 tokens for both. Which is cheaper per task on input alone, and what would change it?","options":[{"text":"Team B, because a shorter prompt always dominates the price difference.","correct":false,"tempting":"Inverts the arithmetic: 400 tokens at 2.00 is 800 microdollars against 280 for A, so the price difference dominates here."},{"text":"They are equivalent, because cost per task depends on output tokens, which are identical.","correct":false,"tempting":"Output is identical, but input differs by a factor of nearly three in cost and is the larger term for both."},{"text":"Team A at 280 microdollars against 800; prefix caching on A's constant examples, or a much higher retry rate for A, would move the comparison.","correct":true},{"text":"Team A, and nothing could change it, since a ten-times price gap cannot be closed by prompt length.","correct":false,"tempting":"A ten-times price gap is closed by a 3.5-times length gap plus retries or caching, so the conclusion is not robust."}]},"prev":{"slug":"measuring-latency-properly","title":"Measuring latency properly","url":"https://addaly.com/learn/open-models/measuring-latency-properly"},"next":{"slug":"swapping-the-model","title":"Swapping the model under a live product","url":"https://addaly.com/learn/open-models/swapping-the-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Cost per task, not cost per token', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/cost-per-task (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Cost per task, not cost per token","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Cost is tokens per task times price times retries, so a cheaper model carrying a long few-shot prompt or a reasoning model spending output tokens on thinking can cost several times more than a dearer model with a short prompt.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"dollars a month, measured rather than estimated","title":"One classification task, a hundred thousand requests a month","series":[{"label":"Reasoning model, 900 output tokens of thinking","value":124,"highlight":true},{"label":"Rented 70B, 400-token prompt","value":94},{"label":"Rented 8B, 900 tokens of examples","value":32},{"label":"Self-hosted tuned 3B, no examples","value":4}],"caption":"The ranking is not the ranking of the price list. The reasoning model’s per-million price looked cheap and it spends nine hundred output tokens thinking, billed at two to four times the input rate. The tuned 3B is cheapest to run and carries a fine-tune’s development cost and permanent maintenance."},"alt":"Figure: One classification task, a hundred thousand requests a month. Bars (dollars a month, measured rather than estimated): Reasoning model, 900 output tokens of thinking 124; Rented 70B, 400-token prompt 94; Rented 8B, 900 tokens of examples 32; Self-hosted tuned 3B, no examples 4. The ranking is not the ranking of the price list. The reasoning model’s per-million price looked cheap and it spends nine hundred output tokens thinking, billed at two to four times the input rate. The tuned 3B is cheapest to run and carries a fine-tune’s development cost and permanent maintenance."}},"blocks":[{"t":"h2","id":"the-price-list-is-not-the-cost","runs":[{"text":"The price list is not the cost"}]},{"t":"p","runs":[{"text":"A model at 15 cents per million tokens is not cheaper than one at 60 cents until you know how many tokens each spends on your task. Three things routinely invert the ranking."}]},{"t":"p","runs":[{"text":"Tokens per task.","b":true},{"text":" A weaker model that needs eight few-shot examples in every prompt carries those examples on every request. A stronger model that works with three sentences of instruction does not. Two thousand tokens of examples at a hundred thousand requests a month is 200 million input tokens that the cheaper model spends and the dearer one does not."}]},{"t":"p","runs":[{"text":"Retries.","b":true},{"text":" A model that produces invalid output 6% of the time costs 1.06 attempts per task, plus the engineering and the latency. At 20% it costs 1.25 attempts and a support problem."}]},{"t":"p","runs":[{"text":"Thinking tokens.","b":true},{"text":" A reasoning model’s chain of thought is billed as output, and output is priced two to four times input. Three to twenty times the output tokens for the same visible answer is the range from the earlier lesson, and it can make a model with an attractive price the most expensive option available."}]},{"t":"h2","id":"the-formula","runs":[{"text":"The formula"}]},{"t":"code","lang":"","label":"Text","text":"cost per task = (input_tokens  x input_price\n               + output_tokens x output_price)\n              x (1 + retry_rate)"},{"t":"p","runs":[{"text":"Everything else is a detail of how you count the first four terms."}]},{"t":"h2","id":"a-worked-comparison","runs":[{"text":"A worked comparison"}]},{"t":"p","runs":[{"text":"A support classification task, 100,000 requests a month. Measured, not guessed:"}]},{"t":"p","runs":[{"text":"Rented 8B, few-shot prompt.","b":true},{"text":" 1,400 input tokens (900 of examples), 60 output. At 20 cents per million in and 40 out: 1,400 × 0.20 + 60 × 0.40 = 304 microdollars per task, about "},{"text":"30 dollars a month","b":true},{"text":". Retry rate 4%, so about 32."}]},{"t":"p","runs":[{"text":"Rented 70B, short prompt.","b":true},{"text":" 400 input, 60 output. At 2.00 in and 2.40 out: 400 × 2.00 + 60 × 2.40 = 944 microdollars, about "},{"text":"94 dollars a month","b":true},{"text":". Retry rate 1%."}]},{"t":"p","runs":[{"text":"Reasoning model, short prompt.","b":true},{"text":" 400 input, 900 output including thinking. Even at the 8B’s prices that is 1,240 microdollars ("},{"text":"124 dollars a month","b":true},{"text":") from a model whose price list looked cheap."}]},{"t":"p","runs":[{"text":"Self-hosted tuned 3B.","b":true},{"text":" No prompt examples needed: 200 input, 60 output. On an always-free CPU instance for a batch workload, or a few dollars a month of a shared GPU. Plus the fine-tune’s development cost and its permanent maintenance."}]},{"t":"p","runs":[{"text":"The ranking here is not the ranking of the price list, and it moves again if the volume is ten times larger or ten times smaller."}]},{"t":"h2","id":"prefix-caching-changes-the-arithmetic","runs":[{"text":"Prefix caching changes the arithmetic"}]},{"t":"p","runs":[{"text":"If every request starts with the same 900 tokens of examples and system prompt, prefix caching means that portion is computed once and reused. Where a provider offers cached-input pricing it is typically a fraction of the standard input price; where you self-host, the saving is compute rather than money but shows up as capacity."}]},{"t":"p","runs":[{"text":"This is the single change that most often makes a few-shot prompt affordable, and it is a configuration flag rather than a project. Structure your prompt so the constant part comes first and the variable part last, or caching cannot help you."}]},{"t":"h2","id":"the-self-hosted-version","runs":[{"text":"The self-hosted version"}]},{"t":"code","lang":"","label":"Text","text":"cost per task = (GPU cost per hour) / (tasks completed per hour)"},{"t":"p","runs":[{"text":"The denominator is where batching lives, and it is why the utilisation argument from the rent-or-own lesson matters here too. The same GPU serving one request at a time and serving thirty concurrently has the same numerator and a wildly different denominator."}]},{"t":"h2","id":"what-to-actually-do","runs":[{"text":"What to actually do"}]},{"t":"p","runs":[{"text":"Instrument it. Log input tokens, output tokens, retries and the model for every request. Then cost per task is a query rather than an estimate, and you can see it change when somebody adds two paragraphs to the prompt."}]},{"t":"code","lang":"json","label":"JSON","text":"{\"task\":\"triage\",\"model\":\"qwen3-8b\",\"prompt_tok\":1412,\n \"completion_tok\":58,\"attempt\":1,\"ms\":840}"},{"t":"p","runs":[{"text":"Six fields. Without them, every cost conversation is a guess, and the first thing anybody does in an optimisation exercise is spend a week collecting exactly this."}]}],"check":{"question":[{"text":"Team A uses a model at 20 cents per million input tokens with a 1,400-token prompt. Team B uses one at 2 dollars per million with a 400-token prompt. Output is 60 tokens for both. Which is cheaper per task on input alone, and what would change it?"}],"options":[{"text":[{"text":"Team B, because a shorter prompt always dominates the price difference."}],"correct":false,"tempting":[{"text":"Inverts the arithmetic: 400 tokens at 2.00 is 800 microdollars against 280 for A, so the price difference dominates here."}]},{"text":[{"text":"They are equivalent, because cost per task depends on output tokens, which are identical."}],"correct":false,"tempting":[{"text":"Output is identical, but input differs by a factor of nearly three in cost and is the larger term for both."}]},{"text":[{"text":"Team A at 280 microdollars against 800; prefix caching on A’s constant examples, or a much higher retry rate for A, would move the comparison."}],"correct":true,"tempting":null},{"text":[{"text":"Team A, and nothing could change it, since a ten-times price gap cannot be closed by prompt length."}],"correct":false,"tempting":[{"text":"A ten-times price gap is closed by a 3.5-times length gap plus retries or caching, so the conclusion is not robust."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"measuring-latency-properly","title":"Measuring latency properly","minutes":8},"next":{"slug":"swapping-the-model","title":"Swapping the model under a live product","minutes":9}}}