{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"cost-at-steady-state","title":"The cost at steady state","url":"https://addaly.com/learn/fine-tuning/cost-at-steady-state","position":79,"of":84,"minutes":9,"answer":"At two million requests a month a self-hosted tuned 8B on redundant cards costs more than a hosted small model before anybody is paid to maintain it.","keyIdea":"At two million requests a month a self-hosted tuned 8B on redundant cards costs more than a hosted small model before anybody is paid to maintain it, so the crossover sits above roughly two million requests or wherever latency, privacy or vendor stability overrides the arithmetic entirely."},"body":{"format":"markdown","text":"## Compute the whole thing, once\n\nThe fine-tune was justified on cost. Here is how to check whether it was right, with everything included.\n\nTake a workload: 2,000,000 requests a month, averaging 1,200 input tokens and 300 output tokens. That is 2.4 billion input and 600 million output tokens.\n\n**Hosted frontier model**, at roughly $3 per million input and $15 per million output:\n\n```\ninput   2,400M × $3/M   = $7,200\noutput    600M × $15/M  = $9,000\n                          -------\n                          $16,200 / month\n```\n\n**Hosted small model**, at roughly $0.15 and $0.60:\n\n```\ninput   2,400M × $0.15/M = $360\noutput    600M × $0.60/M = $360\n                           ------\n                           $720 / month\n```\n\n**Self-hosted tuned 8B.** Two always-on L4-class cards at about $0.80 an hour, because one is a single point of failure:\n\n```\n2 × $0.80 × 730h         = $1,168 / month\n```\n\nNotice the ordering. The self-hosted tuned model is **more expensive than a hosted small model** at this volume, before you have paid anybody to maintain it.\n\n## The costs that do not appear on a cloud bill\n\n- **Idle capacity.** Traffic is not flat. Sized for peak, your cards are idle much of the night, and you pay for every hour.\n- **Redundancy.** One card is a single point of failure, so it is two, which is the single largest multiplier in the calculation.\n- **The engineer.** Deployments, upgrades, a driver incident, an out-of-memory at 2 a.m. Even at a tenth of a person's time this is the biggest line in the table and it is the one that never gets written down.\n- **Retraining.** Each base-model move costs days.\n- **Evaluation maintenance.** The held-out set ages and must be refreshed.\n\nAgainst those, one cost that is often understated on the other side: **hosted models change under you**. A provider deprecates a version or updates it silently, and your carefully measured behaviour shifts. A model you host does exactly what it did yesterday, forever, and for some products that stability is the whole point.\n\n## Where the crossover actually is\n\nRough shape, for a typical workload:\n\n- **Under ~200k requests a month.** Hosted, almost always. The fixed costs dominate everything.\n- **200k to 2M.** Genuinely close. Depends on how spiky the traffic is, whether you already run GPUs, and whether you have someone to maintain it.\n- **Over ~2M, or with a latency or privacy constraint.** Self-hosting wins clearly, and the fixed costs amortise into nothing.\n\nAnd the constraints that override the arithmetic entirely: data that may not leave your infrastructure, an air-gapped deployment, a latency budget under 200 ms, or a per-request cost of zero because you call the model on every keystroke.\n\n> Figure: Two million requests a month, all in. Bars ($ per month): Hosted frontier model 16200; Self-hosted tuned 8B, two cards 1168; Hosted small model 720. The self-hosted tuned model is dearer than a hosted small model at this volume, before anybody is paid to maintain it. Idle capacity, redundancy and the engineer are the lines that never get written down.\n\n## Serverless, the middle option\n\nServerless GPU platforms charge per second of compute with scale-to-zero. For spiky or low-volume traffic they are much cheaper than always-on cards, and you pay for a cold start of several seconds on the first request after idle.\n\nGood for internal tools, batch jobs and anything where a few seconds of latency on the first request is acceptable. Poor for interactive products with sparse traffic, where every user hits a cold start.\n\n## Throughput, which decides how many cards\n\nThe number that turns a workload into a card count is output tokens per second under batching. It varies enormously with model size, quantisation, sequence length, engine and batch size: an 8B model in AWQ on a modern mid-range card with vLLM, well batched, is in the range of several hundred to a couple of thousand output tokens per second aggregate.\n\nThat range is too wide to plan with, which is the point: **measure it on your hardware with your traffic shape** before sizing anything. A load test with realistic prompt lengths and realistic concurrency takes an afternoon and replaces a guess that could be wrong by a factor of four in either direction.\n\n## Write the table down\n\nPut all the options on one page with quality, p95 latency and monthly cost including people. Then make the decision in front of somebody who will be affected by it.\n\nMost of the time the arithmetic is clear once it is written. The reason fine-tuning projects get justified on cost and then cost more is not that the arithmetic is hard. It is that nobody did it."},"check":{"question":"A team justifies self-hosting a tuned 8B model on cost at 500,000 requests a month. Which omitted item most often reverses that conclusion?","options":[{"text":"The one-off training cost, which is larger than the monthly saving at this volume.","correct":false,"tempting":"Training compute for a LoRA is a few pounds; it is the human cost around it, not the GPU time, that dominates."},{"text":"The cost of the evaluation set, which must be relabelled monthly to detect drift.","correct":false,"tempting":"Names a real recurring cost that is periodic rather than monthly and much smaller than continuous operational effort."},{"text":"Redundancy and the engineering time to operate it, which appear on no cloud bill.","correct":true},{"text":"Egress charges on model weights, which are incurred each time an instance restarts and pulls the checkpoint.","correct":false,"tempting":"Picks a genuine but trivial line item; a few gigabytes per restart is negligible against the other terms."}]},"prev":{"slug":"many-adapters-at-once","title":"Serving many adapters from one base","url":"https://addaly.com/learn/fine-tuning/many-adapters-at-once"},"next":{"slug":"monitoring-a-tuned-model","title":"Monitoring a model that fails silently","url":"https://addaly.com/learn/fine-tuning/monitoring-a-tuned-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The cost at steady state', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/cost-at-steady-state (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The cost at steady state","courseTitle":"Fine-Tuning, and When Not To","module":{"index":9,"slug":"shipping-and-living-with-it","title":"Shipping it, and living with it"},"answer":null,"keyIdea":"At two million requests a month a self-hosted tuned 8B on redundant cards costs more than a hosted small model before anybody is paid to maintain it, so the crossover sits above roughly two million requests or wherever latency, privacy or vendor stability overrides the arithmetic entirely.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"$ per month","title":"Two million requests a month, all in","series":[{"label":"Hosted frontier model","value":16200},{"label":"Self-hosted tuned 8B, two cards","value":1168},{"label":"Hosted small model","value":720,"highlight":true}],"caption":"The self-hosted tuned model is dearer than a hosted small model at this volume, before anybody is paid to maintain it. Idle capacity, redundancy and the engineer are the lines that never get written down."},"alt":"Figure: Two million requests a month, all in. Bars ($ per month): Hosted frontier model 16200; Self-hosted tuned 8B, two cards 1168; Hosted small model 720. The self-hosted tuned model is dearer than a hosted small model at this volume, before anybody is paid to maintain it. Idle capacity, redundancy and the engineer are the lines that never get written down."}},"blocks":[{"t":"h2","id":"compute-the-whole-thing-once","runs":[{"text":"Compute the whole thing, once"}]},{"t":"p","runs":[{"text":"The fine-tune was justified on cost. Here is how to check whether it was right, with everything included."}]},{"t":"p","runs":[{"text":"Take a workload: 2,000,000 requests a month, averaging 1,200 input tokens and 300 output tokens. That is 2.4 billion input and 600 million output tokens."}]},{"t":"p","runs":[{"text":"Hosted frontier model","b":true},{"text":", at roughly $3 per million input and $15 per million output:"}]},{"t":"code","lang":"","label":"Text","text":"input   2,400M × $3/M   = $7,200\noutput    600M × $15/M  = $9,000\n                          -------\n                          $16,200 / month"},{"t":"p","runs":[{"text":"Hosted small model","b":true},{"text":", at roughly $0.15 and $0.60:"}]},{"t":"code","lang":"","label":"Text","text":"input   2,400M × $0.15/M = $360\noutput    600M × $0.60/M = $360\n                           ------\n                           $720 / month"},{"t":"p","runs":[{"text":"Self-hosted tuned 8B.","b":true},{"text":" Two always-on L4-class cards at about $0.80 an hour, because one is a single point of failure:"}]},{"t":"code","lang":"","label":"Text","text":"2 × $0.80 × 730h         = $1,168 / month"},{"t":"p","runs":[{"text":"Notice the ordering. The self-hosted tuned model is "},{"text":"more expensive than a hosted small model","b":true},{"text":" at this volume, before you have paid anybody to maintain it."}]},{"t":"h2","id":"the-costs-that-do-not-appear-on-a-cloud-bill","runs":[{"text":"The costs that do not appear on a cloud bill"}]},{"t":"ul","items":[[{"text":"Idle capacity.","b":true},{"text":" Traffic is not flat. Sized for peak, your cards are idle much of the night, and you pay for every hour."}],[{"text":"Redundancy.","b":true},{"text":" One card is a single point of failure, so it is two, which is the single largest multiplier in the calculation."}],[{"text":"The engineer.","b":true},{"text":" Deployments, upgrades, a driver incident, an out-of-memory at 2 a.m. Even at a tenth of a person’s time this is the biggest line in the table and it is the one that never gets written down."}],[{"text":"Retraining.","b":true},{"text":" Each base-model move costs days."}],[{"text":"Evaluation maintenance.","b":true},{"text":" The held-out set ages and must be refreshed."}]]},{"t":"p","runs":[{"text":"Against those, one cost that is often understated on the other side: "},{"text":"hosted models change under you","b":true},{"text":". A provider deprecates a version or updates it silently, and your carefully measured behaviour shifts. A model you host does exactly what it did yesterday, forever, and for some products that stability is the whole point."}]},{"t":"h2","id":"where-the-crossover-actually-is","runs":[{"text":"Where the crossover actually is"}]},{"t":"p","runs":[{"text":"Rough shape, for a typical workload:"}]},{"t":"ul","items":[[{"text":"Under ~200k requests a month.","b":true},{"text":" Hosted, almost always. The fixed costs dominate everything."}],[{"text":"200k to 2M.","b":true},{"text":" Genuinely close. Depends on how spiky the traffic is, whether you already run GPUs, and whether you have someone to maintain it."}],[{"text":"Over ~2M, or with a latency or privacy constraint.","b":true},{"text":" Self-hosting wins clearly, and the fixed costs amortise into nothing."}]]},{"t":"p","runs":[{"text":"And the constraints that override the arithmetic entirely: data that may not leave your infrastructure, an air-gapped deployment, a latency budget under 200 ms, or a per-request cost of zero because you call the model on every keystroke."}]},{"t":"h2","id":"serverless-the-middle-option","runs":[{"text":"Serverless, the middle option"}]},{"t":"p","runs":[{"text":"Serverless GPU platforms charge per second of compute with scale-to-zero. For spiky or low-volume traffic they are much cheaper than always-on cards, and you pay for a cold start of several seconds on the first request after idle."}]},{"t":"p","runs":[{"text":"Good for internal tools, batch jobs and anything where a few seconds of latency on the first request is acceptable. Poor for interactive products with sparse traffic, where every user hits a cold start."}]},{"t":"h2","id":"throughput-which-decides-how-many-cards","runs":[{"text":"Throughput, which decides how many cards"}]},{"t":"p","runs":[{"text":"The number that turns a workload into a card count is output tokens per second under batching. It varies enormously with model size, quantisation, sequence length, engine and batch size: an 8B model in AWQ on a modern mid-range card with vLLM, well batched, is in the range of several hundred to a couple of thousand output tokens per second aggregate."}]},{"t":"p","runs":[{"text":"That range is too wide to plan with, which is the point: "},{"text":"measure it on your hardware with your traffic shape","b":true},{"text":" before sizing anything. A load test with realistic prompt lengths and realistic concurrency takes an afternoon and replaces a guess that could be wrong by a factor of four in either direction."}]},{"t":"h2","id":"write-the-table-down","runs":[{"text":"Write the table down"}]},{"t":"p","runs":[{"text":"Put all the options on one page with quality, p95 latency and monthly cost including people. Then make the decision in front of somebody who will be affected by it."}]},{"t":"p","runs":[{"text":"Most of the time the arithmetic is clear once it is written. The reason fine-tuning projects get justified on cost and then cost more is not that the arithmetic is hard. It is that nobody did it."}]}],"check":{"question":[{"text":"A team justifies self-hosting a tuned 8B model on cost at 500,000 requests a month. Which omitted item most often reverses that conclusion?"}],"options":[{"text":[{"text":"The one-off training cost, which is larger than the monthly saving at this volume."}],"correct":false,"tempting":[{"text":"Training compute for a LoRA is a few pounds; it is the human cost around it, not the GPU time, that dominates."}]},{"text":[{"text":"The cost of the evaluation set, which must be relabelled monthly to detect drift."}],"correct":false,"tempting":[{"text":"Names a real recurring cost that is periodic rather than monthly and much smaller than continuous operational effort."}]},{"text":[{"text":"Redundancy and the engineering time to operate it, which appear on no cloud bill."}],"correct":true,"tempting":null},{"text":[{"text":"Egress charges on model weights, which are incurred each time an instance restarts and pulls the checkpoint."}],"correct":false,"tempting":[{"text":"Picks a genuine but trivial line item; a few gigabytes per restart is negligible against the other terms."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"many-adapters-at-once","title":"Serving many adapters from one base","minutes":8},"next":{"slug":"monitoring-a-tuned-model","title":"Monitoring a model that fails silently","minutes":9}}}