Running your own inference: the break-even arithmetic

Three reasons, and only three

Run your own model when the data cannot leave your infrastructure; when the volume is high and steady enough to keep a card busy most of the day; or when you need something the APIs do not sell: a fine-tuned model of your own, a custom sampling method, latency inside your own network. “It will be cheaper” is not on the list, because at the utilisation most products have, it is not. This lesson is the arithmetic that shows when it becomes true.

The hardware, in numbers

A card with 24 GB of memory (a consumer RTX 4090, or an L4 in the cloud) holds an 8-billion-parameter model at 16-bit precision, or a 14-billion model quantised to 4 bits. An 80 GB card, an A100 or H100, holds a 70-billion model at 4 bits. Rental prices at the cheaper GPU clouds run roughly $0.50 to $1 an hour for a 24 GB card and $2 to $4 an hour for an 80 GB card; the large cloud providers charge more. A month of one 80 GB card at $3 an hour is about $2,200 whether it serves a million requests or none.

Throughput, and the utilisation trap

vLLM, free and open source, serves a model with continuous batching, which packs many concurrent requests into each forward pass. An 8-billion model on one 24 GB card can produce on the order of a thousand output tokens a second aggregated across requests; a 70-billion model at 4 bits on one 80 GB card, a few hundred. Suppose the big card manages 500 tokens a second flat out. That is 1.8 million tokens an hour. At $3 an hour, the card’s cost is about $1.70 per million tokens at full load, comparable to the API price for a 70-billion-class open model, which sits somewhere between $0.50 and $1 per million on most open-model APIs.

Now put in the real utilisation. Traffic is bursty and follows the working day; a card that is busy twenty per cent of the time produces a fifth of the tokens for the same rent. The effective price becomes $8.50 per million, ten times the API. That is the trap, and it is the reason the formula has three terms:

cost per million tokens = hourly price / (tokens per hour at full load × utilisation)

You pay for the hour. The API charges for the token. Until the hours are full, the token is cheaper.

The break-even

For small models the API almost always wins. A 24 GB card at $0.60 an hour producing a million tokens an hour at full load is $0.60 per million; the API sells comparable small models for $0.10 to $0.30, and the card is not at full load. For 70-billion-class models, the card at sustained utilisation above roughly sixty or seventy per cent starts to match or beat the API; below that, no. Autoscaling helps at the margin and brings cold starts of a minute or more while a model loads.

Run your own numbers: your peak and average tokens per hour from the traces, the throughput of the model you want on the card you can afford, the rental price. If the utilisation term comes out under half, the answer is the API unless one of the other two reasons applies.

Quantisation

Storing weights in 4 bits instead of 16 cuts memory four-fold and raises throughput, at a small cost in quality: typically a point or two on general benchmarks, more on arithmetic and code. Eight-bit is nearly lossless. Measure the loss on your evaluation set, not on the model card’s benchmark; the harness from the evaluation module exists for exactly this comparison.

The operations you inherit

Tens of gigabytes of weights to download and store; driver and CUDA versions that must agree with the serving library; out-of-memory crashes when a long context times a burst of concurrency exhausts the cache the model keeps for each active request; monitoring, upgrades, security patches, and someone to be woken up. A team of one should weigh those hours against the rental saving, and usually the hours win.

The free path, and the middle path

For development, Ollama or llama.cpp runs a small quantised model on a laptop, a few tokens a second on a CPU, which is enough to build against and not enough for users. The smallest production step is one rented card with vLLM.

Between the API and the card sits the honest default for “I want an open model”: the same open weights, served per token, by OpenRouter, Together, Groq and others, at token prices with no operations. And serverless GPU platforms bill by the second for self-hosted models that need only occasional bursts. Most teams that think they want to self-host actually want one of these two.

The decision in a sentence

Self-host when the data or the capability forces it, or when the card would be busy most of the day. Otherwise pay per token, and re-run the arithmetic when the traffic doubles.