Renting open weights by the token

A market you should use before you buy hardware

Open weights are served commercially by a large number of companies: Together, Fireworks, DeepInfra, Groq, Cerebras, Novita, Nebius, Hyperbolic, and aggregators such as OpenRouter and Hugging Face Inference Providers that route to them. All of them speak the OpenAI chat format, so one script tests all of them.

This is the fastest path to a decision. Comparing six models across three providers takes an hour and no downloads. Buying a GPU first and then discovering that a 7B does not do your task is the expensive order to do things in.

The thing nobody tells you: the same model is not the same model

Two providers serving Llama-3.3-70B-Instruct can give you noticeably different quality, and neither is lying about which model they run. The differences come from serving decisions:

• Quantisation. One serves bf16, another fp8, another int4. Cheaper prices usually mean more aggressive quantisation, and it is often not stated on the pricing page.
• Context limit. A provider may serve a 128k model with a 32k cap, and silently truncate.
• Default sampling. Different default temperature, top-p and repetition penalty, applied when you do not specify. Always specify.
• A modified system prompt or safety layer inserted ahead of yours.

OpenRouter exposes the quantisation and context length per provider and lets you pin to a specific one, which is the reason to prefer an aggregator that shows this over one that does not. If you benchmark a model on one provider and deploy against another, you have not benchmarked what you deployed.

Price and speed have different shapes

Pricing is per million tokens, quoted separately for input and output, and output is typically two to four times the input price. Across the market, an 8B class model sits in the low tens of cents per million tokens, a 70B in the low dollars, and the very large mixture-of-experts models above that. The spread between the cheapest and dearest provider for the identical model is routinely three to five times.

Speed splits into two numbers that trade against each other. Time to first token is what an interactive user feels; tokens per second is what a batch job cares about. Specialist hardware providers such as Groq and Cerebras deliver output speeds an order of magnitude above GPU providers on supported models, which changes what is possible for anything interactive, at the cost of a much shorter model menu.

For batch work, none of that matters and you should sort by price.

Privacy, which is usually the real question

Policies differ and they are the reason many teams self-host despite the cost. Ask three questions of any provider before sending real data:

1. Are prompts and completions retained, and for how long?
2. Are they used for training, and is opting out the default or a setting?
3. Where does the request physically execute, and does that satisfy your data residency obligations?

Aggregators complicate this because your request passes through them to a third party, and the effective policy is the strictest link in the chain, not the one on the aggregator’s page. Where an aggregator offers a no-logging setting, understand that it constrains which providers can serve you.

Free tiers

Several providers offer free credit or a free tier on smaller models, and Hugging Face gives a monthly credit allowance on inference providers. These are enough to evaluate half a dozen models properly and to run a hobby project. They are not enough to serve a product, and rate limits on free tiers make latency measurements meaningless.

How to use the market

• Evaluate on a provider, deploy where the constraints say. If data may leave, staying rented is usually cheaper than owning until volume is high.
• Pin the provider and the quantisation, and record both next to the model revision in your inventory.
• Re-run your evaluation set after any provider change. A silent switch to a cheaper serving configuration looks exactly like the model getting worse for no reason, and this is a common and confusing production incident.
• Keep the weights anyway if the model matters to your product. The provider may drop it; your copy will not.