The cost at steady state

Compute the whole thing, once

The fine-tune was justified on cost. Here is how to check whether it was right, with everything included.

Take a workload: 2,000,000 requests a month, averaging 1,200 input tokens and 300 output tokens. That is 2.4 billion input and 600 million output tokens.

Hosted frontier model, at roughly $3 per million input and $15 per million output:

input   2,400M × $3/M   = $7,200
output    600M × $15/M  = $9,000
                          -------
                          $16,200 / month

Hosted small model, at roughly $0.15 and $0.60:

input   2,400M × $0.15/M = $360
output    600M × $0.60/M = $360
                           ------
                           $720 / month

Self-hosted tuned 8B. Two always-on L4-class cards at about $0.80 an hour, because one is a single point of failure:

2 × $0.80 × 730h         = $1,168 / month

Notice the ordering. The self-hosted tuned model is more expensive than a hosted small model at this volume, before you have paid anybody to maintain it.

The costs that do not appear on a cloud bill

• Idle capacity. Traffic is not flat. Sized for peak, your cards are idle much of the night, and you pay for every hour.
• Redundancy. One card is a single point of failure, so it is two, which is the single largest multiplier in the calculation.
• The engineer. Deployments, upgrades, a driver incident, an out-of-memory at 2 a.m. Even at a tenth of a person’s time this is the biggest line in the table and it is the one that never gets written down.
• Retraining. Each base-model move costs days.
• Evaluation maintenance. The held-out set ages and must be refreshed.

Against those, one cost that is often understated on the other side: hosted models change under you. A provider deprecates a version or updates it silently, and your carefully measured behaviour shifts. A model you host does exactly what it did yesterday, forever, and for some products that stability is the whole point.

Where the crossover actually is

Rough shape, for a typical workload:

• Under ~200k requests a month. Hosted, almost always. The fixed costs dominate everything.
• 200k to 2M. Genuinely close. Depends on how spiky the traffic is, whether you already run GPUs, and whether you have someone to maintain it.
• Over ~2M, or with a latency or privacy constraint. Self-hosting wins clearly, and the fixed costs amortise into nothing.

And the constraints that override the arithmetic entirely: data that may not leave your infrastructure, an air-gapped deployment, a latency budget under 200 ms, or a per-request cost of zero because you call the model on every keystroke.

Serverless, the middle option

Serverless GPU platforms charge per second of compute with scale-to-zero. For spiky or low-volume traffic they are much cheaper than always-on cards, and you pay for a cold start of several seconds on the first request after idle.

Good for internal tools, batch jobs and anything where a few seconds of latency on the first request is acceptable. Poor for interactive products with sparse traffic, where every user hits a cold start.

Throughput, which decides how many cards

The number that turns a workload into a card count is output tokens per second under batching. It varies enormously with model size, quantisation, sequence length, engine and batch size: an 8B model in AWQ on a modern mid-range card with vLLM, well batched, is in the range of several hundred to a couple of thousand output tokens per second aggregate.

That range is too wide to plan with, which is the point: measure it on your hardware with your traffic shape before sizing anything. A load test with realistic prompt lengths and realistic concurrency takes an afternoon and replaces a guess that could be wrong by a factor of four in either direction.

Write the table down

Put all the options on one page with quality, p95 latency and monthly cost including people. Then make the decision in front of somebody who will be affected by it.

Most of the time the arithmetic is clear once it is written. The reason fine-tuning projects get justified on cost and then cost more is not that the arithmetic is hard. It is that nobody did it.