What it costs, and where the cost hides

Do the arithmetic once, properly

Take a real service: 12,000 runs a month, each averaging six steps. Because every step resends the growing transcript, a run sends about 18,000 input tokens in total and produces 900 output tokens.

Suppose input costs 50 pence per million tokens and output £2 per million; rates change constantly, so treat these as an illustration and substitute your provider’s current numbers.

• Input: 12,000 × 18,000 = 216 million tokens → about £108
• Output: 12,000 × 900 = 10.8 million tokens → about £22

So roughly £130 a month, and against even a few hours of human time that is trivially worth it. Now double the steps. The transcript at each turn grows too, so total input roughly quadruples: about £400 for input (3.7 times). Twice the steps is around four times the cost, and that single relationship explains most surprising bills.

It is also why the advice throughout this course is to shorten the chain rather than reach for a cheaper model. Halving the steps beats a 40 per cent cheaper model, and it usually improves reliability at the same time.

The three drivers, in order

Transcript growth. Long runs, big tool results, whole documents pasted into context. Fixing one chatty tool that returns a thousand lines on every call is frequently the largest single saving available.

Retries. A failure that retries three times costs four runs. A retry storm during a provider incident can cost more in an afternoon than the service does in a month, which is why the budget cap belongs in code and not in a spreadsheet.

Runs that should not have happened. Duplicate webhook deliveries, a scheduler firing twice, a development loop left running over a weekend, a test suite pointed at production credentials. Every team has one of these stories, and the cause is always an absent cap rather than an absent intention.

Where cost hides

Your evaluation suite, if it runs on every commit. Shadow mode, which is pure cost by design. The development environment, where somebody is iterating on a prompt against a 40,000-token fixture. Embedding backfills after a model change. And the long tail of documents ten times larger than the median, which contribute far more than their share of the volume.

Measure them separately, because a bill you cannot decompose is a bill you cannot act on.

Make the bill readable

One API key per automation, per environment. Providers report spend per key, so this single decision converts an opaque monthly total into a line-item breakdown at no cost. Do it before you need it; retrofitting keys after six months means six months of unattributable history.

Then record the cost per run in your own trace store, and put cost per outcome on the dashboard: not “£130 a month” but “9 pence per resolved ticket, against roughly £6 for a human touch”. The second version is the one that survives a budget review, and it is the one that tells you honestly when an automation is not worth it.

Enforce a budget in code

A soft alarm at 60 per cent of the monthly allowance and a hard stop at 100, checked in the runner. Per-tenant caps if one customer can drive your volume. And a per-minute spend ceiling as well as a per-month one, because a parallel job can produce a month’s budget in an hour and a monthly cap will notice far too late.

This is the same principle as budgets and stopping rules: anything you need guaranteed goes in the harness. A cost policy that exists only in a document is a cost policy that will be discovered to have failed on an invoice.

The three levers, in order

1. Shorten the chain. Fewer steps, smaller observations, handles instead of blobs. Biggest return, and it improves reliability too.
2. Cache. Keep the prompt prefix stable so provider caching applies, and cache your own deterministic tool results.
3. Right-size the model. Use a small model for classification and routing sub-steps, keep the large one for the step that needs it. Measure rather than assume; on narrow tasks the gap is often smaller than the price difference.

And a closing honesty: pre-build cost estimates are routinely two to three times low, because nobody models retries, the long tail of large inputs, or the evaluation runs. Estimate, then double it, then measure the first month against the estimate: that comparison is how you get good at the next one.