{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"the-arithmetic-of-tokens-and-gpus","title":"What a prediction costs, worked out","url":"https://addaly.com/learn/cloud-and-shipping/the-arithmetic-of-tokens-and-gpus","position":51,"of":74,"minutes":10,"answer":"Compute cost per request from token prices or from GPU-hours divided by realistic utilisation, and the comparison settles itself: hosted APIs win while you are idle, your own hardware wins only at sustained volume.","keyIdea":"Compute cost per request from token prices or from GPU-hours divided by realistic utilisation, and the comparison settles itself: hosted APIs win while you are idle, your own hardware wins only at sustained volume, and caching plus a per-user quota is the lever you control either way."},"body":{"format":"markdown","text":"## Do the arithmetic before you choose\n\nEvery argument about hosted APIs versus self-hosting is settled by two numbers you can compute in ten minutes: cost per request, and requests per month. People skip this and choose by preference, then discover the answer on an invoice.\n\nPrices change constantly, so learn the *method* and re-run it with today's numbers. The shape of the answer has been stable for years even as the constants fall.\n\n## Cost per request, for a hosted API\n\nProviders bill per million tokens, input and output priced separately, with output typically three to five times the price of input. Take a worked example with round numbers: input at $0.30 per million tokens, output at $1.20 per million.\n\nA request with a 1,200-token prompt and a 400-token answer costs:\n\n- Input: 1,200 ÷ 1,000,000 × $0.30 = $0.00036\n- Output: 400 ÷ 1,000,000 × $1.20 = $0.00048\n- Total: about **$0.0008**, or 0.08 cents.\n\nAt 100,000 requests a month that is $80. At 10 million it is $8,000. The interesting part is what moves it: doubling the answer length adds more than doubling the prompt length, and a retrieval step that adds 3,000 tokens of context to every request roughly triples the input cost. Both are decisions you make in code, not properties of the model.\n\nTwo discounts are worth knowing because they are large and often unused. **Prompt caching** reprices the repeated prefix of a request (a long system prompt or a fixed document) at a fraction of the normal input rate, often around a tenth, when the same prefix is sent again within a short window. **Batch endpoints** halve the price for work that can wait hours. Anything nightly should be on the batch tier.\n\n## Cost per request, for your own GPU\n\nRent, do not buy, until you have a year of data. A mid-range inference GPU rents for roughly $0.40 to $1.50 an hour depending on the card and the provider, and the smaller specialist clouds are consistently cheaper than the large ones for this.\n\nThe key figure is **utilisation**, and it is the one people get wrong. A GPU costs the same when idle. At $0.80 an hour:\n\n- $576 a month, continuously.\n- If it serves 20 requests per second, sustained, that is 52 million requests a month and about $0.00001 each: a hundred times cheaper than the API.\n- If it serves one request every ten seconds (a realistic figure for a small product) that is 260,000 requests a month and about $0.0022 each, nearly three times *more* expensive than the API.\n\nThat is the whole comparison. Self-hosting wins on volume and loses on idleness, and small products are mostly idle. Add to the self-hosted side the engineering time to run it, which is the cost nobody puts in the spreadsheet and often the largest one.\n\n> Figure: The same request, three ways of paying for it. Bars (US dollars per 1,000 requests): Hosted API, 1,200 in and 400 out 0.8; Own GPU, sustained 20 requests a second 0.01; Own GPU, one request every 10 seconds 2.2. A rented GPU at 80 cents an hour costs 576 dollars a month whether it works or idles. Busy, it undercuts the API a hundredfold. At one request every ten seconds, which is a realistic small product, it costs nearly three times more.\n\nThe honest cases for self-hosting are: sustained high volume, data that must not leave your infrastructure, a fine-tuned model that no provider serves, and unmetered experimentation on a machine you already own.\n\n## Caching, which is the biggest lever you control\n\nThe cheapest inference is the one you do not run.\n\n- **Exact-match cache.** Hash the normalised input and store the output. On any product with repeated queries this removes a surprising share (often 10 to 30%) for the cost of a Redis lookup.\n- **Embedding cache.** Embeddings are deterministic for a given model and text. Never compute one twice; store it with the row.\n- **Semantic cache.** Match near-duplicate questions by embedding similarity. Effective and risky: a threshold that is too loose returns the answer to a similar-but-different question. Log every hit and check them for a week before trusting it.\n\n## Put a ceiling on it, per user and per day\n\nThe cost lesson earlier in this course made the general point; models make it urgent, because a single user with a script can spend more in an hour than your monthly hosting. Three controls, all of which belong in code before launch:\n\n1. A **per-user daily quota**, counted in a table, claimed before the call and refunded if the provider fails.\n2. A **hard max-tokens** on every request. Without it, one pathological input can generate the model's maximum output.\n3. A **monthly spend cap** at the provider, plus an alert at half of it. Caps stop money leaving; alerts only tell you it left.\n\nRecord the actual token counts from each response into your own table alongside the user and the feature. Providers report a total; only you can say which feature spent it, and that breakdown is what lets you cut cost without cutting the product.\n\n## The free path, for learning\n\nOllama or llama.cpp on your own laptop costs nothing and teaches every concept here, at a token rate slow enough to make the arithmetic obvious. Free API tiers exist at several providers and are enough for a course project. Nothing in this module requires you to spend money to understand it."},"check":{"question":"A team rents a GPU at $0.80/hour to replace an API costing $0.0008 per request. They serve about 2 requests per minute. What does the arithmetic say?","options":[{"text":"Self-hosting is cheaper, because the fixed cost is spread over unlimited requests","correct":false,"tempting":"Only true if the requests exist; the cost is fixed but the volume here is not, which is precisely the trap."},{"text":"Roughly break-even, since GPU pricing is designed to match API pricing at typical load","correct":false,"tempting":"Assumes providers price against your utilisation. They price against theirs, which is far higher than a small product's."},{"text":"Self-hosting costs about $576 a month for around 86,000 requests, roughly $0.0067 each, about eight times the API price","correct":true},{"text":"Cost is unchanged and the decision should be made on latency alone","correct":false,"tempting":"Latency is a real consideration, but the costs here differ by nearly an order of magnitude, so it is not the deciding factor."}]},"prev":{"slug":"where-inference-time-goes","title":"Where inference time actually goes","url":"https://addaly.com/learn/cloud-and-shipping/where-inference-time-goes"},"next":{"slug":"which-model-answered","title":"Which model, which prompt, which version","url":"https://addaly.com/learn/cloud-and-shipping/which-model-answered"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a prediction costs, worked out', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/the-arithmetic-of-tokens-and-gpus (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a prediction costs, worked out","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"Compute cost per request from token prices or from GPU-hours divided by realistic utilisation, and the comparison settles itself: hosted APIs win while you are idle, your own hardware wins only at sustained volume, and caching plus a per-user quota is the lever you control either way.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"US dollars per 1,000 requests","title":"The same request, three ways of paying for it","series":[{"label":"Hosted API, 1,200 in and 400 out","value":0.8,"highlight":true},{"label":"Own GPU, sustained 20 requests a second","value":0.01},{"label":"Own GPU, one request every 10 seconds","value":2.2}],"caption":"A rented GPU at 80 cents an hour costs 576 dollars a month whether it works or idles. Busy, it undercuts the API a hundredfold. At one request every ten seconds, which is a realistic small product, it costs nearly three times more."},"alt":"Figure: The same request, three ways of paying for it. Bars (US dollars per 1,000 requests): Hosted API, 1,200 in and 400 out 0.8; Own GPU, sustained 20 requests a second 0.01; Own GPU, one request every 10 seconds 2.2. A rented GPU at 80 cents an hour costs 576 dollars a month whether it works or idles. Busy, it undercuts the API a hundredfold. At one request every ten seconds, which is a realistic small product, it costs nearly three times more."}},"blocks":[{"t":"h2","id":"do-the-arithmetic-before-you-choose","runs":[{"text":"Do the arithmetic before you choose"}]},{"t":"p","runs":[{"text":"Every argument about hosted APIs versus self-hosting is settled by two numbers you can compute in ten minutes: cost per request, and requests per month. People skip this and choose by preference, then discover the answer on an invoice."}]},{"t":"p","runs":[{"text":"Prices change constantly, so learn the "},{"text":"method","i":true},{"text":" and re-run it with today’s numbers. The shape of the answer has been stable for years even as the constants fall."}]},{"t":"h2","id":"cost-per-request-for-a-hosted-api","runs":[{"text":"Cost per request, for a hosted API"}]},{"t":"p","runs":[{"text":"Providers bill per million tokens, input and output priced separately, with output typically three to five times the price of input. Take a worked example with round numbers: input at $0.30 per million tokens, output at $1.20 per million."}]},{"t":"p","runs":[{"text":"A request with a 1,200-token prompt and a 400-token answer costs:"}]},{"t":"ul","items":[[{"text":"Input: 1,200 ÷ 1,000,000 × $0.30 = $0.00036"}],[{"text":"Output: 400 ÷ 1,000,000 × $1.20 = $0.00048"}],[{"text":"Total: about "},{"text":"$0.0008","b":true},{"text":", or 0.08 cents."}]]},{"t":"p","runs":[{"text":"At 100,000 requests a month that is $80. At 10 million it is $8,000. The interesting part is what moves it: doubling the answer length adds more than doubling the prompt length, and a retrieval step that adds 3,000 tokens of context to every request roughly triples the input cost. Both are decisions you make in code, not properties of the model."}]},{"t":"p","runs":[{"text":"Two discounts are worth knowing because they are large and often unused. "},{"text":"Prompt caching","b":true},{"text":" reprices the repeated prefix of a request (a long system prompt or a fixed document) at a fraction of the normal input rate, often around a tenth, when the same prefix is sent again within a short window. "},{"text":"Batch endpoints","b":true},{"text":" halve the price for work that can wait hours. Anything nightly should be on the batch tier."}]},{"t":"h2","id":"cost-per-request-for-your-own-gpu","runs":[{"text":"Cost per request, for your own GPU"}]},{"t":"p","runs":[{"text":"Rent, do not buy, until you have a year of data. A mid-range inference GPU rents for roughly $0.40 to $1.50 an hour depending on the card and the provider, and the smaller specialist clouds are consistently cheaper than the large ones for this."}]},{"t":"p","runs":[{"text":"The key figure is "},{"text":"utilisation","b":true},{"text":", and it is the one people get wrong. A GPU costs the same when idle. At $0.80 an hour:"}]},{"t":"ul","items":[[{"text":"$576 a month, continuously."}],[{"text":"If it serves 20 requests per second, sustained, that is 52 million requests a month and about $0.00001 each: a hundred times cheaper than the API."}],[{"text":"If it serves one request every ten seconds (a realistic figure for a small product) that is 260,000 requests a month and about $0.0022 each, nearly three times "},{"text":"more","i":true},{"text":" expensive than the API."}]]},{"t":"p","runs":[{"text":"That is the whole comparison. Self-hosting wins on volume and loses on idleness, and small products are mostly idle. Add to the self-hosted side the engineering time to run it, which is the cost nobody puts in the spreadsheet and often the largest one."}]},{"t":"p","runs":[{"text":"The honest cases for self-hosting are: sustained high volume, data that must not leave your infrastructure, a fine-tuned model that no provider serves, and unmetered experimentation on a machine you already own."}]},{"t":"h2","id":"caching-which-is-the-biggest-lever-you-control","runs":[{"text":"Caching, which is the biggest lever you control"}]},{"t":"p","runs":[{"text":"The cheapest inference is the one you do not run."}]},{"t":"ul","items":[[{"text":"Exact-match cache.","b":true},{"text":" Hash the normalised input and store the output. On any product with repeated queries this removes a surprising share (often 10 to 30%) for the cost of a Redis lookup."}],[{"text":"Embedding cache.","b":true},{"text":" Embeddings are deterministic for a given model and text. Never compute one twice; store it with the row."}],[{"text":"Semantic cache.","b":true},{"text":" Match near-duplicate questions by embedding similarity. Effective and risky: a threshold that is too loose returns the answer to a similar-but-different question. Log every hit and check them for a week before trusting it."}]]},{"t":"h2","id":"put-a-ceiling-on-it-per-user-and-per-day","runs":[{"text":"Put a ceiling on it, per user and per day"}]},{"t":"p","runs":[{"text":"The cost lesson earlier in this course made the general point; models make it urgent, because a single user with a script can spend more in an hour than your monthly hosting. Three controls, all of which belong in code before launch:"}]},{"t":"ol","start":1,"items":[[{"text":"A "},{"text":"per-user daily quota","b":true},{"text":", counted in a table, claimed before the call and refunded if the provider fails."}],[{"text":"A "},{"text":"hard max-tokens","b":true},{"text":" on every request. Without it, one pathological input can generate the model’s maximum output."}],[{"text":"A "},{"text":"monthly spend cap","b":true},{"text":" at the provider, plus an alert at half of it. Caps stop money leaving; alerts only tell you it left."}]]},{"t":"p","runs":[{"text":"Record the actual token counts from each response into your own table alongside the user and the feature. Providers report a total; only you can say which feature spent it, and that breakdown is what lets you cut cost without cutting the product."}]},{"t":"h2","id":"the-free-path-for-learning","runs":[{"text":"The free path, for learning"}],"role":"free"},{"t":"p","runs":[{"text":"Ollama or llama.cpp on your own laptop costs nothing and teaches every concept here, at a token rate slow enough to make the arithmetic obvious. Free API tiers exist at several providers and are enough for a course project. Nothing in this module requires you to spend money to understand it."}]}],"check":{"question":[{"text":"A team rents a GPU at $0.80/hour to replace an API costing $0.0008 per request. They serve about 2 requests per minute. What does the arithmetic say?"}],"options":[{"text":[{"text":"Self-hosting is cheaper, because the fixed cost is spread over unlimited requests"}],"correct":false,"tempting":[{"text":"Only true if the requests exist; the cost is fixed but the volume here is not, which is precisely the trap."}]},{"text":[{"text":"Roughly break-even, since GPU pricing is designed to match API pricing at typical load"}],"correct":false,"tempting":[{"text":"Assumes providers price against your utilisation. They price against theirs, which is far higher than a small product’s."}]},{"text":[{"text":"Self-hosting costs about $576 a month for around 86,000 requests, roughly $0.0067 each, about eight times the API price"}],"correct":true,"tempting":null},{"text":[{"text":"Cost is unchanged and the decision should be made on latency alone"}],"correct":false,"tempting":[{"text":"Latency is a real consideration, but the costs here differ by nearly an order of magnitude, so it is not the deciding factor."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"where-inference-time-goes","title":"Where inference time actually goes","minutes":10},"next":{"slug":"which-model-answered","title":"Which model, which prompt, which version","minutes":9}}}