Renting the compute instead of owning it

The Hub as a router

Many model pages carry an inference widget and a code snippet that calls a hosted API. Behind it sits a router: Hugging Face forwards your request to one of several third-party providers (Together, Fireworks, Replicate, SambaNova, Cerebras and others) under a single token and a single bill.

from huggingface_hub import InferenceClient

client = InferenceClient(api_key=os.environ["HF_TOKEN"])

out = client.chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Summarise this in two lines: ..."}],
    max_tokens=200,
)
print(out.choices[0].message.content)

The interface is OpenAI-compatible, so the openai package pointed at the router’s base URL works unchanged, which matters, because it means switching between a hosted model, a local llama.cpp server and a provider is a change of base URL rather than a rewrite.

There is a small monthly credit on free accounts and a larger one on paid; beyond that you are billed at the provider’s rates with no markup, which is stated on the pricing page and worth re-reading, since these arrangements change.

The four options, and the honest comparison

Serverless through a provider. No infrastructure, per-token billing, cold starts on less popular models, and your input goes to a third party. Right for prototypes, spiky traffic and anything where you would rather not run a GPU.

A dedicated endpoint. Hugging Face Inference Endpoints, or the equivalent on a cloud, gives you a container running your model on hardware you rent by the hour, with autoscaling and scale-to-zero available. Predictable latency, private, and billed for uptime rather than requests: the same trap as paid Spaces, with the same fix: configure scale-to-zero before you configure anything else.

Your own server. vLLM or text-generation-inference on a rented or owned GPU. Cheapest per token at volume, most work, complete control of data.

On the user’s device. transformers.js in a browser, GGUF on a laptop, a small model on a phone. Zero marginal cost, complete privacy, limited to small models. Module three covered it, and it remains the most underused option.

The arithmetic that decides it

The crossover is not where people guess. Take a rough figure of a few tens of US cents per million tokens for a small open model through a provider, and around a dollar an hour for a modest GPU.

At 100,000 tokens a day, serverless costs a few cents a day and a dedicated GPU costs twenty-four dollars. Serverless wins by a very wide margin.

At 100 million tokens a day, serverless is in the tens of dollars a day and one busy GPU is still twenty-four. Now owning wins, provided you can keep it busy, and utilisation is the variable nobody models. A GPU idle 80% of the time costs the same as a GPU at full load, so your effective rate is five times the headline one.

Run the arithmetic with your own volume before choosing. Most teams self-host too early, and a smaller number stay on per-token billing long past the point where a single card would be cheaper.

Privacy is not a footnote here

Sending data to a provider means it leaves your machine. What is logged, for how long, and whether it may be used for training differs by provider and by plan, and the answer lives in their terms rather than in the Hub’s interface. For health, financial, legal or children’s data (or anything a user would not have pasted into a public website) that question has to be answered before the integration is written, not after.

The local path is available and free. A 3B model at Q4_K_M on a CPU handles a great deal of routine work, and nothing leaves the building.

Reliability, stated plainly

A provider can deprecate a model, change its quantization, rate limit you, or go down. All four have happened to somebody this year. Three mitigations cost little:

1. Keep the provider behind one function in your code, so swapping is a small change.
2. Have a fallback, even a smaller local model, for when the primary is unavailable.
3. Cache aggressively. A large share of production requests are repeats, and a hash-keyed cache is often the single biggest cost saving available.

Do this now

Estimate your monthly token volume, then compute what it costs serverless and what one rented GPU costs for the same month. Write both numbers down. Most people have never done this and are surprised by which side they are on.