What a prediction costs, worked out

Do the arithmetic before you choose

Every argument about hosted APIs versus self-hosting is settled by two numbers you can compute in ten minutes: cost per request, and requests per month. People skip this and choose by preference, then discover the answer on an invoice.

Prices change constantly, so learn the method and re-run it with today’s numbers. The shape of the answer has been stable for years even as the constants fall.

Cost per request, for a hosted API

Providers bill per million tokens, input and output priced separately, with output typically three to five times the price of input. Take a worked example with round numbers: input at $0.30 per million tokens, output at $1.20 per million.

A request with a 1,200-token prompt and a 400-token answer costs:

• Input: 1,200 ÷ 1,000,000 × $0.30 = $0.00036
• Output: 400 ÷ 1,000,000 × $1.20 = $0.00048
• Total: about $0.0008, or 0.08 cents.

At 100,000 requests a month that is $80. At 10 million it is $8,000. The interesting part is what moves it: doubling the answer length adds more than doubling the prompt length, and a retrieval step that adds 3,000 tokens of context to every request roughly triples the input cost. Both are decisions you make in code, not properties of the model.

Two discounts are worth knowing because they are large and often unused. Prompt caching reprices the repeated prefix of a request (a long system prompt or a fixed document) at a fraction of the normal input rate, often around a tenth, when the same prefix is sent again within a short window. Batch endpoints halve the price for work that can wait hours. Anything nightly should be on the batch tier.

Cost per request, for your own GPU

Rent, do not buy, until you have a year of data. A mid-range inference GPU rents for roughly $0.40 to $1.50 an hour depending on the card and the provider, and the smaller specialist clouds are consistently cheaper than the large ones for this.

The key figure is utilisation, and it is the one people get wrong. A GPU costs the same when idle. At $0.80 an hour:

• $576 a month, continuously.
• If it serves 20 requests per second, sustained, that is 52 million requests a month and about $0.00001 each: a hundred times cheaper than the API.
• If it serves one request every ten seconds (a realistic figure for a small product) that is 260,000 requests a month and about $0.0022 each, nearly three times more expensive than the API.

That is the whole comparison. Self-hosting wins on volume and loses on idleness, and small products are mostly idle. Add to the self-hosted side the engineering time to run it, which is the cost nobody puts in the spreadsheet and often the largest one.

The honest cases for self-hosting are: sustained high volume, data that must not leave your infrastructure, a fine-tuned model that no provider serves, and unmetered experimentation on a machine you already own.

Caching, which is the biggest lever you control

The cheapest inference is the one you do not run.

• Exact-match cache. Hash the normalised input and store the output. On any product with repeated queries this removes a surprising share (often 10 to 30%) for the cost of a Redis lookup.
• Embedding cache. Embeddings are deterministic for a given model and text. Never compute one twice; store it with the row.
• Semantic cache. Match near-duplicate questions by embedding similarity. Effective and risky: a threshold that is too loose returns the answer to a similar-but-different question. Log every hit and check them for a week before trusting it.

Put a ceiling on it, per user and per day

The cost lesson earlier in this course made the general point; models make it urgent, because a single user with a script can spend more in an hour than your monthly hosting. Three controls, all of which belong in code before launch:

1. A per-user daily quota, counted in a table, claimed before the call and refunded if the provider fails.
2. A hard max-tokens on every request. Without it, one pathological input can generate the model’s maximum output.
3. A monthly spend cap at the provider, plus an alert at half of it. Caps stop money leaving; alerts only tell you it left.

Record the actual token counts from each response into your own table alongside the user and the feature. Providers report a total; only you can say which feature spent it, and that breakdown is what lets you cut cost without cutting the product.

The free path, for learning

Ollama or llama.cpp on your own laptop costs nothing and teaches every concept here, at a token rate slow enough to make the arithmetic obvious. Free API tiers exist at several providers and are enough for a course project. Nothing in this module requires you to spend money to understand it.