{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"electricity-and-heat","title":"Electricity, heat and the power limit knob","url":"https://addaly.com/learn/running-models-yourself/electricity-and-heat","position":67,"of":82,"minutes":8,"answer":"Electricity is close to a rounding error per token when the card is busy and the dominant waste when it idles, so measure watts at the wall, cap board power for a disproportionate saving.","keyIdea":"Electricity is close to a rounding error per token when the card is busy and the dominant waste when it idles, so measure watts at the wall, cap board power for a disproportionate saving, and check the throttle reasons before blaming software for a gradual loss of speed."},"body":{"format":"markdown","text":"## Work out your own number, because the range is enormous\n\nElectricity is the one input where a figure quoted on the internet is almost certainly wrong for you. Domestic tariffs differ by more than a factor of five between countries, and within a country they differ by time of day, by supplier and by whether you are on a commercial connection. So the method matters and the example number does not.\n\nThe method:\n\n```\nkWh per hour            = system watts / 1000\ncost per hour           = kWh per hour x your tariff per kWh\ncost per million tokens = cost per hour / (tokens per second x 3600 / 1e6)\n```\n\nMeasure system watts at the wall, not from the card's specification. A plug-in energy monitor costs very little and settles the argument: the card's board power is one part of a figure that also includes the CPU, the drives, the fans and the power supply's own losses. A machine with a 350W card commonly draws 450 to 550W under load.\n\nA worked case, with stated assumptions: 500W at the wall, a tariff of 0.20 per kWh, and 1,200 tokens per second sustained. That is 0.10 per hour and 4.32 million tokens per hour, so electricity contributes about **0.023 per million tokens**. Which tells you something useful: when the machine is busy, electricity is close to a rounding error next to the capital cost. Its significance rises exactly as duty cycle falls, because idle power is paid for with no tokens against it.\n\n## Idle draw is the line item that surprises people\n\nA desktop with a large GPU sitting idle typically draws 60 to 120W at the wall, and it does so for the 98% of the time nobody is asking it anything. At 0.20 per kWh, 90W continuously is about 15 currency units a month for producing nothing at all. That figure is frequently larger than the entire API bill it was supposed to replace.\n\nThe fix is unglamorous: suspend the machine when it is not in use, or accept the cold start and unload the model. A machine that wakes on a network packet and loads a model in eight seconds is a better deal than one that idles all year.\n\n## The power limit is free money\n\nGPU power scales worse than linearly with clock speed, so the last slice of performance costs a disproportionate share of the watts. Capping the board power gives back most of those watts for a small loss of throughput:\n\n```bash\nnvidia-smi -q -d POWER | grep -E 'Power Limit'    # see current and range\nsudo nvidia-smi -pl 250                            # cap at 250W\n```\n\nThe usual shape of the result on a large consumer card: capping at roughly 70% of the default limit costs somewhere in the region of 5 to 10% of tokens per second. Measure it yourself with a fixed benchmark before and after, because it varies by card and by workload: decode is memory-bound and loses less than prefill, which is compute-bound and loses more.\n\nThe side effects are all good: lower temperatures, quieter fans, less throttling, and a power supply with more headroom. On a machine that runs a model most of the day, this is one command that pays for itself.\n\n## Heat is a second bill in a hot climate\n\nEvery watt the machine consumes becomes heat in the room. A 500W machine is a 500W heater running continuously, which is pleasant in a cold winter and an entirely different proposition in a room at 35 degrees.\n\nIf you are cooling that room, you pay twice. Air conditioning does not consume as much energy as the heat it removes (a unit with a coefficient of performance around 3 removes roughly three units of heat per unit of electricity) so the rule of thumb is that cooling adds something like a third again to the machine's electricity cost. For a readership where a great deal of this work happens in rooms without climate control at all, the practical consequences are more direct: the card reaches its thermal limit and reduces its own clocks.\n\nThat is worth recognising when it happens, because it looks like the model getting slower for no reason:\n\n```bash\nnvidia-smi -q -d PERFORMANCE | grep -A6 'Clocks Event Reasons'\n# SW Thermal Slowdown : Active   <- the card is protecting itself\n```\n\nA throttled card can lose a quarter of its throughput and never report an error. Check this before concluding an engine upgrade made things worse.\n\n## Laptops are a different machine under load\n\nA laptop GPU's sustained performance is set by the chassis, not the chip. The first minute is fast, the tenth is whatever the cooling allows, and on battery many machines cut the GPU power limit sharply regardless of settings. Benchmark on battery and on mains separately, and benchmark for ten minutes rather than thirty seconds, or your numbers describe a machine you do not have."},"check":{"question":"Capping a card's board power at roughly 70% of its default limit typically costs only a few per cent of tokens per second. What mechanism explains the poor return on those last watts?","options":[{"text":"The driver reserves the top of the power range for display output, so it is not available to compute anyway.","correct":false,"tempting":"Invents a reservation; display output on a headless inference card consumes a negligible and roughly constant amount of power."},{"text":"Quantised models use integer units that run at a fixed clock, so they are insensitive to the power limit.","correct":false,"tempting":"Quantised inference still dequantises and computes in floating point on the same units, and the effect holds for unquantised models too."},{"text":"The cap only applies to idle power states, so peak generation is unaffected.","correct":false,"tempting":"Misreads what the limit does: it constrains the board's sustained draw under load, which is precisely when generation happens."},{"text":"Power scales worse than linearly with clock speed, and decode is memory-bound anyway.","correct":true}]},"prev":{"slug":"utilisation-decides-everything","title":"Duty cycle, and why an idle card is expensive","url":"https://addaly.com/learn/running-models-yourself/utilisation-decides-everything"},"next":{"slug":"buying-the-card","title":"Buying hardware, in the right order","url":"https://addaly.com/learn/running-models-yourself/buying-the-card"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Electricity, heat and the power limit knob', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/electricity-and-heat (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Electricity, heat and the power limit knob","courseTitle":"Running Models Yourself","module":{"index":8,"slug":"what-it-actually-costs","title":"What it actually costs"},"answer":null,"keyIdea":"Electricity is close to a rounding error per token when the card is busy and the dominant waste when it idles, so measure watts at the wall, cap board power for a disproportionate saving, and check the throttle reasons before blaming software for a gradual loss of speed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"work-out-your-own-number-because-the-range-is-enormous","runs":[{"text":"Work out your own number, because the range is enormous"}]},{"t":"p","runs":[{"text":"Electricity is the one input where a figure quoted on the internet is almost certainly wrong for you. Domestic tariffs differ by more than a factor of five between countries, and within a country they differ by time of day, by supplier and by whether you are on a commercial connection. So the method matters and the example number does not."}]},{"t":"p","runs":[{"text":"The method:"}]},{"t":"code","lang":"","label":"Text","text":"kWh per hour            = system watts / 1000\ncost per hour           = kWh per hour x your tariff per kWh\ncost per million tokens = cost per hour / (tokens per second x 3600 / 1e6)"},{"t":"p","runs":[{"text":"Measure system watts at the wall, not from the card’s specification. A plug-in energy monitor costs very little and settles the argument: the card’s board power is one part of a figure that also includes the CPU, the drives, the fans and the power supply’s own losses. A machine with a 350W card commonly draws 450 to 550W under load."}]},{"t":"p","runs":[{"text":"A worked case, with stated assumptions: 500W at the wall, a tariff of 0.20 per kWh, and 1,200 tokens per second sustained. That is 0.10 per hour and 4.32 million tokens per hour, so electricity contributes about "},{"text":"0.023 per million tokens","b":true},{"text":". Which tells you something useful: when the machine is busy, electricity is close to a rounding error next to the capital cost. Its significance rises exactly as duty cycle falls, because idle power is paid for with no tokens against it."}]},{"t":"h2","id":"idle-draw-is-the-line-item-that-surprises-people","runs":[{"text":"Idle draw is the line item that surprises people"}]},{"t":"p","runs":[{"text":"A desktop with a large GPU sitting idle typically draws 60 to 120W at the wall, and it does so for the 98% of the time nobody is asking it anything. At 0.20 per kWh, 90W continuously is about 15 currency units a month for producing nothing at all. That figure is frequently larger than the entire API bill it was supposed to replace."}]},{"t":"p","runs":[{"text":"The fix is unglamorous: suspend the machine when it is not in use, or accept the cold start and unload the model. A machine that wakes on a network packet and loads a model in eight seconds is a better deal than one that idles all year."}]},{"t":"h2","id":"the-power-limit-is-free-money","runs":[{"text":"The power limit is free money"}]},{"t":"p","runs":[{"text":"GPU power scales worse than linearly with clock speed, so the last slice of performance costs a disproportionate share of the watts. Capping the board power gives back most of those watts for a small loss of throughput:"}]},{"t":"code","lang":"bash","label":"Bash","text":"nvidia-smi -q -d POWER | grep -E 'Power Limit'    # see current and range\nsudo nvidia-smi -pl 250                            # cap at 250W"},{"t":"p","runs":[{"text":"The usual shape of the result on a large consumer card: capping at roughly 70% of the default limit costs somewhere in the region of 5 to 10% of tokens per second. Measure it yourself with a fixed benchmark before and after, because it varies by card and by workload: decode is memory-bound and loses less than prefill, which is compute-bound and loses more."}]},{"t":"p","runs":[{"text":"The side effects are all good: lower temperatures, quieter fans, less throttling, and a power supply with more headroom. On a machine that runs a model most of the day, this is one command that pays for itself."}]},{"t":"h2","id":"heat-is-a-second-bill-in-a-hot-climate","runs":[{"text":"Heat is a second bill in a hot climate"}]},{"t":"p","runs":[{"text":"Every watt the machine consumes becomes heat in the room. A 500W machine is a 500W heater running continuously, which is pleasant in a cold winter and an entirely different proposition in a room at 35 degrees."}]},{"t":"p","runs":[{"text":"If you are cooling that room, you pay twice. Air conditioning does not consume as much energy as the heat it removes (a unit with a coefficient of performance around 3 removes roughly three units of heat per unit of electricity) so the rule of thumb is that cooling adds something like a third again to the machine’s electricity cost. For a readership where a great deal of this work happens in rooms without climate control at all, the practical consequences are more direct: the card reaches its thermal limit and reduces its own clocks."}]},{"t":"p","runs":[{"text":"That is worth recognising when it happens, because it looks like the model getting slower for no reason:"}]},{"t":"code","lang":"bash","label":"Bash","text":"nvidia-smi -q -d PERFORMANCE | grep -A6 'Clocks Event Reasons'\n# SW Thermal Slowdown : Active   <- the card is protecting itself"},{"t":"p","runs":[{"text":"A throttled card can lose a quarter of its throughput and never report an error. Check this before concluding an engine upgrade made things worse."}]},{"t":"h2","id":"laptops-are-a-different-machine-under-load","runs":[{"text":"Laptops are a different machine under load"}]},{"t":"p","runs":[{"text":"A laptop GPU’s sustained performance is set by the chassis, not the chip. The first minute is fast, the tenth is whatever the cooling allows, and on battery many machines cut the GPU power limit sharply regardless of settings. Benchmark on battery and on mains separately, and benchmark for ten minutes rather than thirty seconds, or your numbers describe a machine you do not have."}]}],"check":{"question":[{"text":"Capping a card’s board power at roughly 70% of its default limit typically costs only a few per cent of tokens per second. What mechanism explains the poor return on those last watts?"}],"options":[{"text":[{"text":"The driver reserves the top of the power range for display output, so it is not available to compute anyway."}],"correct":false,"tempting":[{"text":"Invents a reservation; display output on a headless inference card consumes a negligible and roughly constant amount of power."}]},{"text":[{"text":"Quantised models use integer units that run at a fixed clock, so they are insensitive to the power limit."}],"correct":false,"tempting":[{"text":"Quantised inference still dequantises and computes in floating point on the same units, and the effect holds for unquantised models too."}]},{"text":[{"text":"The cap only applies to idle power states, so peak generation is unaffected."}],"correct":false,"tempting":[{"text":"Misreads what the limit does: it constrains the board’s sustained draw under load, which is precisely when generation happens."}]},{"text":[{"text":"Power scales worse than linearly with clock speed, and decode is memory-bound anyway."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"utilisation-decides-everything","title":"Duty cycle, and why an idle card is expensive","minutes":9},"next":{"slug":"buying-the-card","title":"Buying hardware, in the right order","minutes":9}}}