Electricity, heat and the power limit knob

Work out your own number, because the range is enormous

Electricity is the one input where a figure quoted on the internet is almost certainly wrong for you. Domestic tariffs differ by more than a factor of five between countries, and within a country they differ by time of day, by supplier and by whether you are on a commercial connection. So the method matters and the example number does not.

The method:

kWh per hour            = system watts / 1000
cost per hour           = kWh per hour x your tariff per kWh
cost per million tokens = cost per hour / (tokens per second x 3600 / 1e6)

Measure system watts at the wall, not from the card’s specification. A plug-in energy monitor costs very little and settles the argument: the card’s board power is one part of a figure that also includes the CPU, the drives, the fans and the power supply’s own losses. A machine with a 350W card commonly draws 450 to 550W under load.

A worked case, with stated assumptions: 500W at the wall, a tariff of 0.20 per kWh, and 1,200 tokens per second sustained. That is 0.10 per hour and 4.32 million tokens per hour, so electricity contributes about 0.023 per million tokens. Which tells you something useful: when the machine is busy, electricity is close to a rounding error next to the capital cost. Its significance rises exactly as duty cycle falls, because idle power is paid for with no tokens against it.

Idle draw is the line item that surprises people

A desktop with a large GPU sitting idle typically draws 60 to 120W at the wall, and it does so for the 98% of the time nobody is asking it anything. At 0.20 per kWh, 90W continuously is about 15 currency units a month for producing nothing at all. That figure is frequently larger than the entire API bill it was supposed to replace.

The fix is unglamorous: suspend the machine when it is not in use, or accept the cold start and unload the model. A machine that wakes on a network packet and loads a model in eight seconds is a better deal than one that idles all year.

The power limit is free money

GPU power scales worse than linearly with clock speed, so the last slice of performance costs a disproportionate share of the watts. Capping the board power gives back most of those watts for a small loss of throughput:

nvidia-smi -q -d POWER | grep -E 'Power Limit'    # see current and range
sudo nvidia-smi -pl 250                            # cap at 250W

The usual shape of the result on a large consumer card: capping at roughly 70% of the default limit costs somewhere in the region of 5 to 10% of tokens per second. Measure it yourself with a fixed benchmark before and after, because it varies by card and by workload: decode is memory-bound and loses less than prefill, which is compute-bound and loses more.

The side effects are all good: lower temperatures, quieter fans, less throttling, and a power supply with more headroom. On a machine that runs a model most of the day, this is one command that pays for itself.

Heat is a second bill in a hot climate

Every watt the machine consumes becomes heat in the room. A 500W machine is a 500W heater running continuously, which is pleasant in a cold winter and an entirely different proposition in a room at 35 degrees.

If you are cooling that room, you pay twice. Air conditioning does not consume as much energy as the heat it removes (a unit with a coefficient of performance around 3 removes roughly three units of heat per unit of electricity) so the rule of thumb is that cooling adds something like a third again to the machine’s electricity cost. For a readership where a great deal of this work happens in rooms without climate control at all, the practical consequences are more direct: the card reaches its thermal limit and reduces its own clocks.

That is worth recognising when it happens, because it looks like the model getting slower for no reason:

nvidia-smi -q -d PERFORMANCE | grep -A6 'Clocks Event Reasons'
# SW Thermal Slowdown : Active   <- the card is protecting itself

A throttled card can lose a quarter of its throughput and never report an error. Check this before concluding an engine upgrade made things worse.

Laptops are a different machine under load

A laptop GPU’s sustained performance is set by the chassis, not the chip. The first minute is fast, the tenth is whatever the cooling allows, and on battery many machines cut the GPU power limit sharply regardless of settings. Benchmark on battery and on mains separately, and benchmark for ten minutes rather than thirty seconds, or your numbers describe a machine you do not have.