Duty cycle, and why an idle card is expensive

The denominator nobody measures

Cost per token has a shape that is easy to state and easy to get wrong:

cost per token = cost per hour of owning the machine / tokens produced in that hour

Everybody argues about the numerator (the price of the card, the electricity tariff) and almost nobody measures the denominator. Yet the denominator moves the answer by a factor of ten or more, while the numerator moves it by tens of per cent.

The reason is that the fixed costs run whether or not you are generating anything. Amortisation runs on the calendar. A resident model holds its VRAM at three in the morning. The machine draws idle power all weekend. An API bills you for tokens and nothing else; your local machine bills you for time and gives you tokens only when you ask.

A worked example that changes minds

Take a workstation whose all-in fixed cost is 100 currency units a month: capital amortised, electricity, the share of the internet connection, whatever your numbers are. Suppose the engine sustains 1,200 tokens per second with full batches.

At 100% duty cycle the machine produces 1200 x 3600 x 24 x 30 tokens in a month, which is about 3,110 million. That is roughly 0.032 per million tokens. Cheaper than any hosted model of comparable size, by a wide margin.

At 2% duty cycle (a small team using a chat interface during working hours, which is a realistic figure) the same machine produces about 62 million tokens a month. That is 1.61 per million tokens. Fifty times more, from exactly the same hardware, running exactly as fast.

Nothing about the model changed. Nothing about the card changed. The only thing that changed is how much of the time it was doing work, and that single number is the difference between local being obviously cheap and obviously expensive.

The tokens you did not generate are the ones you paid the most for.

Measure your duty cycle rather than guessing

You need two numbers: total tokens actually served in a period, and the machine’s sustained peak. The first comes from the gateway, which is already counting for billing:

# vLLM exposes cumulative counters; sample them a day apart
curl -s http://127.0.0.1:8000/metrics | grep -E 'generation_tokens_total|prompt_tokens_total'

The second comes from a saturating load test, the one from the performance module, run at a concurrency high enough that the queue never empties. Then:

duty cycle = tokens per day / (peak tokens per second x 86400)

People’s estimates of this are consistently too high, usually by an order of magnitude. A developer who feels they use the model constantly is typically under 5%, because thinking, reading and typing take far longer than generation does.

The three ways to raise it

Batch the work that can wait. Nightly document summarisation, backfilling embeddings, re-classifying an archive, generating test data, all of it can run at full batch overnight, and all of it is free throughput on a machine you are paying for anyway. This is the single largest lever, and it turns 2% into 30% without buying anything.

Put more people on the same endpoint. Ten colleagues at 2% each is a 20% duty cycle, and continuous batching means they largely do not slow each other down until the card saturates. Sharing is not a courtesy here; it is the economics.

Stop paying for idle. Unload the model after a period of inactivity so the VRAM and the idle power go back: Ollama’s OLLAMA_KEEP_ALIVE does this by default. The cost is a cold start on the next request. On a rented machine, stopping the instance entirely is the version of this that actually saves money.

The trap in the other direction

Raising duty cycle by inventing work is not a saving. A team that starts summarising every document “because the GPU is free” has converted an idle asset into a busy one without producing anything anybody reads. The metric to keep alongside duty cycle is how many of the tokens produced were used by a person or a system that needed them.

There is also a ceiling you cannot cross. A single interactive user never reaches peak throughput, because peak throughput requires many sequences in flight; one user gets single-stream speed, which on a memory-bandwidth-bound decode is a fraction of the batched figure. So the 100% column in the example above is only reachable with concurrency or batch work. For one person, alone, the honest ceiling is far lower, which is most of the answer to why a personal GPU is rarely a cost saving and usually something else: privacy, latency, offline capability, or the pleasure of owning the thing.