{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"utilisation-decides-everything","title":"Duty cycle, and why an idle card is expensive","url":"https://addaly.com/learn/running-models-yourself/utilisation-decides-everything","position":66,"of":82,"minutes":9,"answer":"Cost per token is fixed cost divided by tokens actually produced.","keyIdea":"Cost per token is fixed cost divided by tokens actually produced, so duty cycle moves the answer by a factor of ten while card price and electricity move it by tens of per cent: measure it from the gateway's counters and a saturating load test, and raise it with batch work rather than invented work."},"body":{"format":"markdown","text":"## The denominator nobody measures\n\nCost per token has a shape that is easy to state and easy to get wrong:\n\n```\ncost per token = cost per hour of owning the machine / tokens produced in that hour\n```\n\nEverybody argues about the numerator (the price of the card, the electricity tariff) and almost nobody measures the denominator. Yet the denominator moves the answer by a factor of ten or more, while the numerator moves it by tens of per cent.\n\nThe reason is that the fixed costs run whether or not you are generating anything. Amortisation runs on the calendar. A resident model holds its VRAM at three in the morning. The machine draws idle power all weekend. An API bills you for tokens and nothing else; your local machine bills you for time and gives you tokens only when you ask.\n\n## A worked example that changes minds\n\nTake a workstation whose all-in fixed cost is 100 currency units a month: capital amortised, electricity, the share of the internet connection, whatever your numbers are. Suppose the engine sustains 1,200 tokens per second with full batches.\n\nAt **100% duty cycle** the machine produces 1200 x 3600 x 24 x 30 tokens in a month, which is about 3,110 million. That is roughly **0.032 per million tokens**. Cheaper than any hosted model of comparable size, by a wide margin.\n\n> Figure: Cost per token is fixed cost over tokens actually produced. Line chart of Cost per million tokens against Duty cycle, per cent of the month spent generating: A machine costing 100 a month at 1,200 tok/s (2, 1.61) (5, 0.64) (10, 0.32) (20, 0.16) (50, 0.06) (100, 0.03). Everybody argues about the numerator, the price of the card and the tariff, and almost nobody measures the denominator, which moves the answer by a factor of fifty. Estimates of duty cycle are consistently an order of magnitude too high, because thinking, reading and typing take far longer than generation does.\n\nAt **2% duty cycle** (a small team using a chat interface during working hours, which is a realistic figure) the same machine produces about 62 million tokens a month. That is **1.61 per million tokens**. Fifty times more, from exactly the same hardware, running exactly as fast.\n\nNothing about the model changed. Nothing about the card changed. The only thing that changed is how much of the time it was doing work, and that single number is the difference between local being obviously cheap and obviously expensive.\n\n> The tokens you did not generate are the ones you paid the most for.\n\n## Measure your duty cycle rather than guessing\n\nYou need two numbers: total tokens actually served in a period, and the machine's sustained peak. The first comes from the gateway, which is already counting for billing:\n\n```bash\n# vLLM exposes cumulative counters; sample them a day apart\ncurl -s http://127.0.0.1:8000/metrics | grep -E 'generation_tokens_total|prompt_tokens_total'\n```\n\nThe second comes from a saturating load test, the one from the performance module, run at a concurrency high enough that the queue never empties. Then:\n\n```\nduty cycle = tokens per day / (peak tokens per second x 86400)\n```\n\nPeople's estimates of this are consistently too high, usually by an order of magnitude. A developer who feels they use the model constantly is typically under 5%, because thinking, reading and typing take far longer than generation does.\n\n## The three ways to raise it\n\n**Batch the work that can wait.** Nightly document summarisation, backfilling embeddings, re-classifying an archive, generating test data, all of it can run at full batch overnight, and all of it is free throughput on a machine you are paying for anyway. This is the single largest lever, and it turns 2% into 30% without buying anything.\n\n**Put more people on the same endpoint.** Ten colleagues at 2% each is a 20% duty cycle, and continuous batching means they largely do not slow each other down until the card saturates. Sharing is not a courtesy here; it is the economics.\n\n**Stop paying for idle.** Unload the model after a period of inactivity so the VRAM and the idle power go back: Ollama's `OLLAMA_KEEP_ALIVE` does this by default. The cost is a cold start on the next request. On a rented machine, stopping the instance entirely is the version of this that actually saves money.\n\n## The trap in the other direction\n\nRaising duty cycle by inventing work is not a saving. A team that starts summarising every document \"because the GPU is free\" has converted an idle asset into a busy one without producing anything anybody reads. The metric to keep alongside duty cycle is how many of the tokens produced were used by a person or a system that needed them.\n\nThere is also a ceiling you cannot cross. A single interactive user never reaches peak throughput, because peak throughput requires many sequences in flight; one user gets single-stream speed, which on a memory-bandwidth-bound decode is a fraction of the batched figure. So the 100% column in the example above is only reachable with concurrency or batch work. For one person, alone, the honest ceiling is far lower, which is most of the answer to why a personal GPU is rarely a cost saving and usually something else: privacy, latency, offline capability, or the pleasure of owning the thing."},"check":{"question":"Two teams own identical machines with identical fixed monthly costs. One reports 0.04 per million tokens, the other 1.60. Both measured correctly. What is the most likely explanation?","options":[{"text":"The second team's duty cycle is far lower, so the same fixed cost is divided across far fewer tokens.","correct":true},{"text":"The first team is running a much smaller quantised model, so it produces tokens far faster on the same hardware.","correct":false,"tempting":"Quantisation changes throughput by a factor of two or three, nowhere near the forty-fold gap described here."},{"text":"The second team is including electricity in its figure and the first is not.","correct":false,"tempting":"Electricity is a modest share of fixed cost at any realistic tariff, so omitting it cannot produce a forty-fold difference."},{"text":"The first team is measuring peak throughput rather than sustained throughput, which overstates tokens produced.","correct":false,"tempting":"Peak versus sustained is a real reporting error, but the gap it creates is roughly two-fold, not forty-fold."}]},"prev":{"slug":"the-honest-cost-comparison","title":"The honest cost comparison","url":"https://addaly.com/learn/running-models-yourself/the-honest-cost-comparison"},"next":{"slug":"electricity-and-heat","title":"Electricity, heat and the power limit knob","url":"https://addaly.com/learn/running-models-yourself/electricity-and-heat"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Duty cycle, and why an idle card is expensive', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/utilisation-decides-everything (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Duty cycle, and why an idle card is expensive","courseTitle":"Running Models Yourself","module":{"index":8,"slug":"what-it-actually-costs","title":"What it actually costs"},"answer":null,"keyIdea":"Cost per token is fixed cost divided by tokens actually produced, so duty cycle moves the answer by a factor of ten while card price and electricity move it by tens of per cent: measure it from the gateway’s counters and a saturating load test, and raise it with batch work rather than invented work.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Cost per token is fixed cost over tokens actually produced","series":[{"label":"A machine costing 100 a month at 1,200 tok/s","points":[[2,1.61],[5,0.64],[10,0.32],[20,0.16],[50,0.06],[100,0.03]]}],"xLabel":"Duty cycle, per cent of the month spent generating","yLabel":"Cost per million tokens","caption":"Everybody argues about the numerator, the price of the card and the tariff, and almost nobody measures the denominator, which moves the answer by a factor of fifty. Estimates of duty cycle are consistently an order of magnitude too high, because thinking, reading and typing take far longer than generation does."},"alt":"Figure: Cost per token is fixed cost over tokens actually produced. Line chart of Cost per million tokens against Duty cycle, per cent of the month spent generating: A machine costing 100 a month at 1,200 tok/s (2, 1.61) (5, 0.64) (10, 0.32) (20, 0.16) (50, 0.06) (100, 0.03). Everybody argues about the numerator, the price of the card and the tariff, and almost nobody measures the denominator, which moves the answer by a factor of fifty. Estimates of duty cycle are consistently an order of magnitude too high, because thinking, reading and typing take far longer than generation does."}},"blocks":[{"t":"h2","id":"the-denominator-nobody-measures","runs":[{"text":"The denominator nobody measures"}]},{"t":"p","runs":[{"text":"Cost per token has a shape that is easy to state and easy to get wrong:"}]},{"t":"code","lang":"","label":"Text","text":"cost per token = cost per hour of owning the machine / tokens produced in that hour"},{"t":"p","runs":[{"text":"Everybody argues about the numerator (the price of the card, the electricity tariff) and almost nobody measures the denominator. Yet the denominator moves the answer by a factor of ten or more, while the numerator moves it by tens of per cent."}]},{"t":"p","runs":[{"text":"The reason is that the fixed costs run whether or not you are generating anything. Amortisation runs on the calendar. A resident model holds its VRAM at three in the morning. The machine draws idle power all weekend. An API bills you for tokens and nothing else; your local machine bills you for time and gives you tokens only when you ask."}]},{"t":"h2","id":"a-worked-example-that-changes-minds","runs":[{"text":"A worked example that changes minds"}]},{"t":"p","runs":[{"text":"Take a workstation whose all-in fixed cost is 100 currency units a month: capital amortised, electricity, the share of the internet connection, whatever your numbers are. Suppose the engine sustains 1,200 tokens per second with full batches."}]},{"t":"p","runs":[{"text":"At "},{"text":"100% duty cycle","b":true},{"text":" the machine produces 1200 x 3600 x 24 x 30 tokens in a month, which is about 3,110 million. That is roughly "},{"text":"0.032 per million tokens","b":true},{"text":". Cheaper than any hosted model of comparable size, by a wide margin."}]},{"t":"p","runs":[{"text":"At "},{"text":"2% duty cycle","b":true},{"text":" (a small team using a chat interface during working hours, which is a realistic figure) the same machine produces about 62 million tokens a month. That is "},{"text":"1.61 per million tokens","b":true},{"text":". Fifty times more, from exactly the same hardware, running exactly as fast."}]},{"t":"p","runs":[{"text":"Nothing about the model changed. Nothing about the card changed. The only thing that changed is how much of the time it was doing work, and that single number is the difference between local being obviously cheap and obviously expensive."}]},{"t":"quote","runs":[{"text":"The tokens you did not generate are the ones you paid the most for."}]},{"t":"h2","id":"measure-your-duty-cycle-rather-than-guessing","runs":[{"text":"Measure your duty cycle rather than guessing"}]},{"t":"p","runs":[{"text":"You need two numbers: total tokens actually served in a period, and the machine’s sustained peak. The first comes from the gateway, which is already counting for billing:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# vLLM exposes cumulative counters; sample them a day apart\ncurl -s http://127.0.0.1:8000/metrics | grep -E 'generation_tokens_total|prompt_tokens_total'"},{"t":"p","runs":[{"text":"The second comes from a saturating load test, the one from the performance module, run at a concurrency high enough that the queue never empties. Then:"}]},{"t":"code","lang":"","label":"Text","text":"duty cycle = tokens per day / (peak tokens per second x 86400)"},{"t":"p","runs":[{"text":"People’s estimates of this are consistently too high, usually by an order of magnitude. A developer who feels they use the model constantly is typically under 5%, because thinking, reading and typing take far longer than generation does."}]},{"t":"h2","id":"the-three-ways-to-raise-it","runs":[{"text":"The three ways to raise it"}]},{"t":"p","runs":[{"text":"Batch the work that can wait.","b":true},{"text":" Nightly document summarisation, backfilling embeddings, re-classifying an archive, generating test data, all of it can run at full batch overnight, and all of it is free throughput on a machine you are paying for anyway. This is the single largest lever, and it turns 2% into 30% without buying anything."}]},{"t":"p","runs":[{"text":"Put more people on the same endpoint.","b":true},{"text":" Ten colleagues at 2% each is a 20% duty cycle, and continuous batching means they largely do not slow each other down until the card saturates. Sharing is not a courtesy here; it is the economics."}]},{"t":"p","runs":[{"text":"Stop paying for idle.","b":true},{"text":" Unload the model after a period of inactivity so the VRAM and the idle power go back: Ollama’s "},{"text":"OLLAMA_KEEP_ALIVE","code":true},{"text":" does this by default. The cost is a cold start on the next request. On a rented machine, stopping the instance entirely is the version of this that actually saves money."}]},{"t":"h2","id":"the-trap-in-the-other-direction","runs":[{"text":"The trap in the other direction"}]},{"t":"p","runs":[{"text":"Raising duty cycle by inventing work is not a saving. A team that starts summarising every document “because the GPU is free” has converted an idle asset into a busy one without producing anything anybody reads. The metric to keep alongside duty cycle is how many of the tokens produced were used by a person or a system that needed them."}]},{"t":"p","runs":[{"text":"There is also a ceiling you cannot cross. A single interactive user never reaches peak throughput, because peak throughput requires many sequences in flight; one user gets single-stream speed, which on a memory-bandwidth-bound decode is a fraction of the batched figure. So the 100% column in the example above is only reachable with concurrency or batch work. For one person, alone, the honest ceiling is far lower, which is most of the answer to why a personal GPU is rarely a cost saving and usually something else: privacy, latency, offline capability, or the pleasure of owning the thing."}]}],"check":{"question":[{"text":"Two teams own identical machines with identical fixed monthly costs. One reports 0.04 per million tokens, the other 1.60. Both measured correctly. What is the most likely explanation?"}],"options":[{"text":[{"text":"The second team’s duty cycle is far lower, so the same fixed cost is divided across far fewer tokens."}],"correct":true,"tempting":null},{"text":[{"text":"The first team is running a much smaller quantised model, so it produces tokens far faster on the same hardware."}],"correct":false,"tempting":[{"text":"Quantisation changes throughput by a factor of two or three, nowhere near the forty-fold gap described here."}]},{"text":[{"text":"The second team is including electricity in its figure and the first is not."}],"correct":false,"tempting":[{"text":"Electricity is a modest share of fixed cost at any realistic tariff, so omitting it cannot produce a forty-fold difference."}]},{"text":[{"text":"The first team is measuring peak throughput rather than sustained throughput, which overstates tokens produced."}],"correct":false,"tempting":[{"text":"Peak versus sustained is a real reporting error, but the gap it creates is roughly two-fold, not forty-fold."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"the-honest-cost-comparison","title":"The honest cost comparison","minutes":10},"next":{"slug":"electricity-and-heat","title":"Electricity, heat and the power limit knob","minutes":8}}}