Renting a GPU, and the meter that runs while you think

A different unit of account

Owning a card costs money per month regardless of what you do. Renting one costs money per minute the instance exists, not per minute it computes. That single difference reshapes the engineering, because the thing to minimise is wall-clock time with the meter running, and most of that time is usually not generation.

Where rented hours actually go, in the order people are surprised by:

• Downloading weights. A 40GB model at 200 MB/s is about three and a half minutes, and on a slower link it is twenty. You pay for every one of them, on every cold start, unless the weights are on a persistent volume.
• Building the environment. Installing an engine and compiling kernels can take longer than the job.
• Thinking. An instance left running while you read the output, edit a prompt and try again bills continuously.
• Forgetting. The single largest rental bill anyone reports is an instance nobody shut down.

The discipline that makes it cheap

Treat a rented GPU as a batch machine, even when the work feels interactive. The pattern:

#!/usr/bin/env bash
set -euo pipefail
# weights on a persistent volume, so this is a no-op after the first run
export HF_HOME=/workspace/hf
huggingface-cli download "$MODEL" --local-dir-use-symlinks False
python run_job.py --in /workspace/in.jsonl --out /workspace/out.jsonl
aws s3 cp /workspace/out.jsonl "$RESULTS_URI"   # or any object store
sudo shutdown -h now                             # the important line

The last line is the one people leave out. An instance that terminates itself cannot be forgotten. Develop the job against a small model on your laptop, or against a few rows, and send the full run to the rented machine only when it is known to work.

Keep a persistent volume for the weights and the environment. Storage while the instance is stopped is billed at a small fraction of the compute rate, and it converts a twenty-minute cold start into a one-minute one. Check the egress policy too: pulling results out is usually cheap or free, but some providers charge for it and a large generated dataset is not small.

Interruptible instances

Spot or interruptible capacity is substantially cheaper than on-demand, and it can be reclaimed with little warning. That is an easy trade for the right shape of work and a bad one otherwise.

Suitable: anything idempotent and chunked. Write results incrementally, record which inputs are done, and make a restart resume rather than repeat. Unsuitable: an interactive service where somebody notices the interruption, or a long single operation with no checkpoint.

The honest arithmetic is that an interruptible instance at half the price which loses a quarter of its work to interruptions is still cheaper, but only if restarting is automatic. If restarting costs you ten minutes of attention each time, you have traded currency for your own hours, which the next lesson prices.

Renting versus buying, as one line

months to break even = card price / (hourly rate x hours used per month - running cost per month)

With a card at 1,000 units, a rental at 0.40 per hour, 100 hours of use a month and 10 units a month of electricity, that is 1000 / (40 - 10) = about 33 months. At 400 hours a month it drops to about 6.5 months. So the honest form of the question is not “is renting cheaper” but “how many hours a month will I genuinely use it”, and that number is the one people overestimate.

Two adjustments make the comparison fair. Subtract the card’s expected resale value from its price, which the next lesson covers. And add, to the rental side, the minutes lost to cold starts, because they are billed.

The free tier

For learning, several platforms give real GPU time at no cost: hosted notebook services typically offer a limited number of GPU hours a week, with session timeouts and no persistent storage between sessions. They are entirely adequate for working through this course, for quantising a model, for a small fine-tune, and for benchmarking. They are not suitable for serving anything, because sessions end and there is no stable address.

Use them deliberately: mount your own storage for anything you want to keep, and assume the machine disappears when you close the tab.

What renting costs that owning does not

The privacy argument for local inference does not survive a rented machine unless you trust the operator. Your weights sit on their disk, your prompts pass through their memory, and the instance is administered by someone else. For open weights this is usually fine. For prompts containing data that legally cannot leave your jurisdiction, renting is the same category of decision as using a hosted API, and should be made with the same care, including checking which country the hardware is physically in, which is not always the country in the provider’s address.