The cost model of a feature, line by line

From a call to a feature

The first module priced one request: input tokens at one rate, output at a higher one, cached input at a discount. A feature is that arithmetic multiplied by behaviour (how often people use it, how long their sessions run, how much context each turn carries) plus costs that do not scale with users at all. Write it as a spreadsheet before the feature exists, with guessed numbers, and replace each guess with a measurement from the traces as they arrive. The free path is any spreadsheet; the discipline is filling in every cell.

The variables

Per turn: uncached input tokens, cached input tokens, output tokens. Per session: turns. Per user: sessions per day. Then the population: active users per day. And three prices, from the provider’s page: uncached input, cached input, output.

Turn cost is the sum of the three token counts times their prices. Multiply up.

A worked example

A support assistant over product documentation. Ten thousand active users; each has three sessions a week, so about 4,300 sessions a day; six turns a session. Each turn sends a 1,500-token system prompt and retrieved context that is stable across the session, so cached after the first turn; 1,000 tokens of conversation and new retrieval that are not cached; and receives 300 tokens of answer. Prices: $3 per million uncached input, $0.30 cached, $15 output.

Per turn, after the first in a session: cached 1,500 tokens at $0.30 per million is $0.00045; uncached 1,000 at $3 is $0.003; output 300 at $15 is $0.0045. About $0.008. The first turn of a session pays full price on the prefix, adding about $0.004 once per session.

Per session: six turns at $0.008 plus the first-turn premium, roughly $0.05. Per day: 4,300 sessions is about $220. Per month: around $6,500. Per user per month: sixty-five cents.

What the example teaches

Look at where the money went. Output tokens were more than half of each turn’s cost, at 300 tokens. Uncached input was most of the rest. The cached prefix, which is the largest block of tokens in the request, was almost free. Three consequences follow, and they generalise.

Output length is the first lever. A prompt change that trims answers from 300 tokens to 200 saves a third of the output cost, which is nearly a fifth of the whole feature. That change costs nothing and it is the reason the harness in the evaluation module tracks output length as a regression.

Cache hit rate is the second. The example assumed the prefix was cached on every turn after the first. If sessions are far apart and the cache expires between them, or the prefix changes because retrieved context is placed before the system prompt, the cached line becomes uncached and the turn cost rises by half. The prompt-caching lesson’s ordering rule is worth real money here.

Model choice is the third, and it is multiplicative on everything. The same feature on a model at a tenth of the price costs a tenth, if the evaluation set says it is good enough. The routing lesson later in this block is about paying the high price only where it is needed.

The costs that do not scale with users

Ingestion and embedding of the corpus: small, from the retrieval module, and repeated on re-index. The evaluation harness: tens of dollars a month. The judge on shadow-mode comparisons: more, during a change. Human review of moderation and feedback queues: a person’s hours, and the largest fixed cost most teams have. Write these down too; a feature at a thousand users is dominated by them, and at a hundred thousand they vanish.

Sensitivity

Change one variable at a time by half and double, and watch the monthly figure. Whichever moves it most is the one to instrument first and engineer first. In the example it is output length, then cache hit rate, then turns per session. In a document-analysis feature it is input length. In an inline completion feature, from the interface module, it is calls per user, by a wide margin.

Margin, and the price of the feature

If the feature is charged for, the per-user monthly cost is the floor of the price. If it is free, it is a marketing expense with a known size, and the question is what it earns in retention or conversion. Either way the number should be on a page someone reviews monthly against the traces, because the guesses that built the spreadsheet will be wrong in both directions, and the day the actual figure is twice the model is the day to find out which cell was wrong.