Rent, or own: the arithmetic

Three ways to buy inference

Per token, from an inference provider. You pay for what you use, there is nothing to operate, and the price already includes their batching efficiency.

Per hour, renting a GPU from a cloud. You pay whether or not anything is running.

Outright, buying hardware. You pay once, plus electricity, plus your time.

The decision is arithmetic plus three things arithmetic cannot capture. Do the arithmetic first, because it eliminates most cases.

The numbers, at the time of writing

Per-token prices for open models sit roughly at a few tens of cents per million tokens for an 8B class model and a few dollars per million for a 70B, with output priced two to four times input, and a three to five times spread between providers for the identical model.

An A100 with 80 GB rents for somewhere between one and two dollars an hour depending on the provider and commitment, which is roughly 700 to 1,500 dollars a month running continuously. A consumer card such as a used 3090 with 24 GB costs a few hundred to about a thousand to buy, and draws 300 to 450 watts under load: at typical electricity prices, ten to thirty dollars a month if it runs all day.

The breakeven, and why it usually favours renting

Take a 70B model at roughly two dollars per million output tokens rented per token, against an 80 GB GPU at about 1,100 dollars a month. Breakeven is around 550 million output tokens a month.

That is a very large number. It is roughly 18 million tokens a day, or about 7,000 substantial responses an hour, every hour, all month. Most products are two orders of magnitude below it.

And that comparison assumes you keep the GPU busy, which is the assumption that fails. A GPU at 5% utilisation costs twenty times its sticker price per token delivered. Real traffic is bursty: quiet at night, peaked in the afternoon. A rented endpoint absorbs that for free because someone else’s traffic fills the gaps. Your own machine does not.

So the honest default is: rent per token until you have steady, high, predictable load. The exceptions are not about cost.

The three reasons that are not arithmetic

Data cannot leave. Health records, legal files, recordings of conversations, anything under a data residency obligation or a contract you signed. This is the most common genuine reason to self-host and it does not need a cost justification, though it is worth stating the cost honestly so the decision is made with open eyes.

It must work offline. A factory floor, a field device, a clinic with intermittent connectivity, a laptop on a plane. No amount of price advantage helps a service that is unreachable.

Nobody can take it away. A rented model can be deprecated, reconfigured, price-changed or restricted. Weights on your disk cannot. For a product whose behaviour must be stable for years, this is a real form of insurance.

There is a fourth that people undercount: learning. Running the stack yourself teaches you what the previous six lessons describe. That has value even when it is not the cheapest option.

The costs people leave out

When someone presents a spreadsheet showing self-hosting is cheaper, check for these:

• Utilisation. Divide by the fraction of the month the machine is actually working. This is usually the whole error.
• Redundancy. One GPU is a single point of failure. Two doubles the cost.
• The person. Somebody has to patch, monitor, respond at 3am and handle upgrades. A fraction of an engineer costs more than the hardware.
• Model refresh. A better model arrives every few months. Re-quantising, re-evaluating and re-deploying is recurring work.
• Idle time in development. The machine you bought for production is also the one people leave a notebook running on.

A decision rule you can use today

1. If data may not leave, self-host and stop reading.
2. If it must work offline, self-host and stop reading.
3. Otherwise, estimate monthly tokens honestly, multiply by a provider’s price, and compare against a rented GPU at your realistic utilisation, not at 100%.
4. If renting per token is cheaper, rent, and revisit at ten times the volume.
5. Buy hardware for learning, for a workstation, or for constant load, not to save money on a service that runs for two hours a day.