Cost per task, not cost per token

The price list is not the cost

A model at 15 cents per million tokens is not cheaper than one at 60 cents until you know how many tokens each spends on your task. Three things routinely invert the ranking.

Tokens per task. A weaker model that needs eight few-shot examples in every prompt carries those examples on every request. A stronger model that works with three sentences of instruction does not. Two thousand tokens of examples at a hundred thousand requests a month is 200 million input tokens that the cheaper model spends and the dearer one does not.

Retries. A model that produces invalid output 6% of the time costs 1.06 attempts per task, plus the engineering and the latency. At 20% it costs 1.25 attempts and a support problem.

Thinking tokens. A reasoning model’s chain of thought is billed as output, and output is priced two to four times input. Three to twenty times the output tokens for the same visible answer is the range from the earlier lesson, and it can make a model with an attractive price the most expensive option available.

The formula

cost per task = (input_tokens  x input_price
               + output_tokens x output_price)
              x (1 + retry_rate)

Everything else is a detail of how you count the first four terms.

A worked comparison

A support classification task, 100,000 requests a month. Measured, not guessed:

Rented 8B, few-shot prompt. 1,400 input tokens (900 of examples), 60 output. At 20 cents per million in and 40 out: 1,400 × 0.20 + 60 × 0.40 = 304 microdollars per task, about 30 dollars a month. Retry rate 4%, so about 32.

Rented 70B, short prompt. 400 input, 60 output. At 2.00 in and 2.40 out: 400 × 2.00 + 60 × 2.40 = 944 microdollars, about 94 dollars a month. Retry rate 1%.

Reasoning model, short prompt. 400 input, 900 output including thinking. Even at the 8B’s prices that is 1,240 microdollars (124 dollars a month) from a model whose price list looked cheap.

Self-hosted tuned 3B. No prompt examples needed: 200 input, 60 output. On an always-free CPU instance for a batch workload, or a few dollars a month of a shared GPU. Plus the fine-tune’s development cost and its permanent maintenance.

The ranking here is not the ranking of the price list, and it moves again if the volume is ten times larger or ten times smaller.

Prefix caching changes the arithmetic

If every request starts with the same 900 tokens of examples and system prompt, prefix caching means that portion is computed once and reused. Where a provider offers cached-input pricing it is typically a fraction of the standard input price; where you self-host, the saving is compute rather than money but shows up as capacity.

This is the single change that most often makes a few-shot prompt affordable, and it is a configuration flag rather than a project. Structure your prompt so the constant part comes first and the variable part last, or caching cannot help you.

The self-hosted version

cost per task = (GPU cost per hour) / (tasks completed per hour)

The denominator is where batching lives, and it is why the utilisation argument from the rent-or-own lesson matters here too. The same GPU serving one request at a time and serving thirty concurrently has the same numerator and a wildly different denominator.

What to actually do

Instrument it. Log input tokens, output tokens, retries and the model for every request. Then cost per task is a query rather than an estimate, and you can see it change when somebody adds two paragraphs to the prompt.

{"task":"triage","model":"qwen3-8b","prompt_tok":1412,
 "completion_tok":58,"attempt":1,"ms":840}

Six fields. Without them, every cost conversation is a guess, and the first thing anybody does in an optimisation exercise is spend a week collecting exactly this.