{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"inference-providers","title":"Renting open weights by the token","url":"https://addaly.com/learn/open-models/inference-providers","position":20,"of":71,"minutes":8,"answer":"Providers serving the same open model differ in quantisation, context cap and default sampling.","keyIdea":"Providers serving the same open model differ in quantisation, context cap and default sampling, so a benchmark run against one provider does not transfer to another unless you pin the provider and record its serving configuration."},"body":{"format":"markdown","text":"## A market you should use before you buy hardware\n\nOpen weights are served commercially by a large number of companies: Together, Fireworks, DeepInfra, Groq, Cerebras, Novita, Nebius, Hyperbolic, and aggregators such as OpenRouter and Hugging Face Inference Providers that route to them. All of them speak the OpenAI chat format, so one script tests all of them.\n\nThis is the fastest path to a decision. Comparing six models across three providers takes an hour and no downloads. Buying a GPU first and then discovering that a 7B does not do your task is the expensive order to do things in.\n\n## The thing nobody tells you: the same model is not the same model\n\nTwo providers serving `Llama-3.3-70B-Instruct` can give you noticeably different quality, and neither is lying about which model they run. The differences come from serving decisions:\n\n- **Quantisation.** One serves bf16, another fp8, another int4. Cheaper prices usually mean more aggressive quantisation, and it is often not stated on the pricing page.\n- **Context limit.** A provider may serve a 128k model with a 32k cap, and silently truncate.\n- **Default sampling.** Different default temperature, top-p and repetition penalty, applied when you do not specify. Always specify.\n- **A modified system prompt or safety layer** inserted ahead of yours.\n\nOpenRouter exposes the quantisation and context length per provider and lets you pin to a specific one, which is the reason to prefer an aggregator that shows this over one that does not. If you benchmark a model on one provider and deploy against another, you have not benchmarked what you deployed.\n\n## Price and speed have different shapes\n\nPricing is per million tokens, quoted separately for input and output, and output is typically two to four times the input price. Across the market, an 8B class model sits in the low tens of cents per million tokens, a 70B in the low dollars, and the very large mixture-of-experts models above that. The spread between the cheapest and dearest provider for the identical model is routinely three to five times.\n\nSpeed splits into two numbers that trade against each other. **Time to first token** is what an interactive user feels; **tokens per second** is what a batch job cares about. Specialist hardware providers such as Groq and Cerebras deliver output speeds an order of magnitude above GPU providers on supported models, which changes what is possible for anything interactive, at the cost of a much shorter model menu.\n\nFor batch work, none of that matters and you should sort by price.\n\n## Privacy, which is usually the real question\n\nPolicies differ and they are the reason many teams self-host despite the cost. Ask three questions of any provider before sending real data:\n\n1. Are prompts and completions retained, and for how long?\n2. Are they used for training, and is opting out the default or a setting?\n3. Where does the request physically execute, and does that satisfy your data residency obligations?\n\nAggregators complicate this because your request passes through them to a third party, and the effective policy is the strictest link in the chain, not the one on the aggregator's page. Where an aggregator offers a no-logging setting, understand that it constrains which providers can serve you.\n\n## Free tiers\n\nSeveral providers offer free credit or a free tier on smaller models, and Hugging Face gives a monthly credit allowance on inference providers. These are enough to evaluate half a dozen models properly and to run a hobby project. They are not enough to serve a product, and rate limits on free tiers make latency measurements meaningless.\n\n## How to use the market\n\n- **Evaluate on a provider, deploy where the constraints say.** If data may leave, staying rented is usually cheaper than owning until volume is high.\n- **Pin the provider and the quantisation**, and record both next to the model revision in your inventory.\n- **Re-run your evaluation set after any provider change.** A silent switch to a cheaper serving configuration looks exactly like the model getting worse for no reason, and this is a common and confusing production incident.\n- **Keep the weights anyway** if the model matters to your product. The provider may drop it; your copy will not."},"check":{"question":"A team's summarisation quality drops noticeably one week with no change to their code, prompts or model name. Their aggregator routes to whichever provider is cheapest. What is the most probable cause?","options":[{"text":"The model publisher pushed an updated version under the same name.","correct":false,"tempting":"Possible in principle, but published open weights are versioned repositories and providers pin them; routing is the far more common cause."},{"text":"Their input documents drifted, and the model is unchanged.","correct":false,"tempting":"Worth checking, but a step change in one week with the same document pipeline points at the serving side."},{"text":"Routing moved them to a provider serving the same model at a lower precision or a shorter context cap.","correct":true},{"text":"Aggregate demand raised latency, and slower generation produces lower-quality output.","correct":false,"tempting":"Confuses speed with quality; the same weights and sampler produce the same distribution regardless of how long it takes."}]},"prev":{"slug":"what-each-size-can-do","title":"What a model of each size can honestly do","url":"https://addaly.com/learn/open-models/what-each-size-can-do"},"next":{"slug":"base-instruct-reasoning","title":"Base, Instruct, Reasoning","url":"https://addaly.com/learn/open-models/base-instruct-reasoning"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Renting open weights by the token', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/inference-providers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Renting open weights by the token","courseTitle":"The Open Model Ecosystem","module":{"index":2,"slug":"the-map-of-the-ecosystem","title":"The map, and how to read it"},"answer":null,"keyIdea":"Providers serving the same open model differ in quantisation, context cap and default sampling, so a benchmark run against one provider does not transfer to another unless you pin the provider and record its serving configuration.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-market-you-should-use-before-you-buy-hardware","runs":[{"text":"A market you should use before you buy hardware"}]},{"t":"p","runs":[{"text":"Open weights are served commercially by a large number of companies: Together, Fireworks, DeepInfra, Groq, Cerebras, Novita, Nebius, Hyperbolic, and aggregators such as OpenRouter and Hugging Face Inference Providers that route to them. All of them speak the OpenAI chat format, so one script tests all of them."}]},{"t":"p","runs":[{"text":"This is the fastest path to a decision. Comparing six models across three providers takes an hour and no downloads. Buying a GPU first and then discovering that a 7B does not do your task is the expensive order to do things in."}]},{"t":"h2","id":"the-thing-nobody-tells-you-the-same-model-is-not-the-same-mo","runs":[{"text":"The thing nobody tells you: the same model is not the same model"}]},{"t":"p","runs":[{"text":"Two providers serving "},{"text":"Llama-3.3-70B-Instruct","code":true},{"text":" can give you noticeably different quality, and neither is lying about which model they run. The differences come from serving decisions:"}]},{"t":"ul","items":[[{"text":"Quantisation.","b":true},{"text":" One serves bf16, another fp8, another int4. Cheaper prices usually mean more aggressive quantisation, and it is often not stated on the pricing page."}],[{"text":"Context limit.","b":true},{"text":" A provider may serve a 128k model with a 32k cap, and silently truncate."}],[{"text":"Default sampling.","b":true},{"text":" Different default temperature, top-p and repetition penalty, applied when you do not specify. Always specify."}],[{"text":"A modified system prompt or safety layer","b":true},{"text":" inserted ahead of yours."}]]},{"t":"p","runs":[{"text":"OpenRouter exposes the quantisation and context length per provider and lets you pin to a specific one, which is the reason to prefer an aggregator that shows this over one that does not. If you benchmark a model on one provider and deploy against another, you have not benchmarked what you deployed."}]},{"t":"h2","id":"price-and-speed-have-different-shapes","runs":[{"text":"Price and speed have different shapes"}]},{"t":"p","runs":[{"text":"Pricing is per million tokens, quoted separately for input and output, and output is typically two to four times the input price. Across the market, an 8B class model sits in the low tens of cents per million tokens, a 70B in the low dollars, and the very large mixture-of-experts models above that. The spread between the cheapest and dearest provider for the identical model is routinely three to five times."}]},{"t":"p","runs":[{"text":"Speed splits into two numbers that trade against each other. "},{"text":"Time to first token","b":true},{"text":" is what an interactive user feels; "},{"text":"tokens per second","b":true},{"text":" is what a batch job cares about. Specialist hardware providers such as Groq and Cerebras deliver output speeds an order of magnitude above GPU providers on supported models, which changes what is possible for anything interactive, at the cost of a much shorter model menu."}]},{"t":"p","runs":[{"text":"For batch work, none of that matters and you should sort by price."}]},{"t":"h2","id":"privacy-which-is-usually-the-real-question","runs":[{"text":"Privacy, which is usually the real question"}]},{"t":"p","runs":[{"text":"Policies differ and they are the reason many teams self-host despite the cost. Ask three questions of any provider before sending real data:"}]},{"t":"ol","start":1,"items":[[{"text":"Are prompts and completions retained, and for how long?"}],[{"text":"Are they used for training, and is opting out the default or a setting?"}],[{"text":"Where does the request physically execute, and does that satisfy your data residency obligations?"}]]},{"t":"p","runs":[{"text":"Aggregators complicate this because your request passes through them to a third party, and the effective policy is the strictest link in the chain, not the one on the aggregator’s page. Where an aggregator offers a no-logging setting, understand that it constrains which providers can serve you."}]},{"t":"h2","id":"free-tiers","runs":[{"text":"Free tiers"}]},{"t":"p","runs":[{"text":"Several providers offer free credit or a free tier on smaller models, and Hugging Face gives a monthly credit allowance on inference providers. These are enough to evaluate half a dozen models properly and to run a hobby project. They are not enough to serve a product, and rate limits on free tiers make latency measurements meaningless."}]},{"t":"h2","id":"how-to-use-the-market","runs":[{"text":"How to use the market"}]},{"t":"ul","items":[[{"text":"Evaluate on a provider, deploy where the constraints say.","b":true},{"text":" If data may leave, staying rented is usually cheaper than owning until volume is high."}],[{"text":"Pin the provider and the quantisation","b":true},{"text":", and record both next to the model revision in your inventory."}],[{"text":"Re-run your evaluation set after any provider change.","b":true},{"text":" A silent switch to a cheaper serving configuration looks exactly like the model getting worse for no reason, and this is a common and confusing production incident."}],[{"text":"Keep the weights anyway","b":true},{"text":" if the model matters to your product. The provider may drop it; your copy will not."}]]}],"check":{"question":[{"text":"A team’s summarisation quality drops noticeably one week with no change to their code, prompts or model name. Their aggregator routes to whichever provider is cheapest. What is the most probable cause?"}],"options":[{"text":[{"text":"The model publisher pushed an updated version under the same name."}],"correct":false,"tempting":[{"text":"Possible in principle, but published open weights are versioned repositories and providers pin them; routing is the far more common cause."}]},{"text":[{"text":"Their input documents drifted, and the model is unchanged."}],"correct":false,"tempting":[{"text":"Worth checking, but a step change in one week with the same document pipeline points at the serving side."}]},{"text":[{"text":"Routing moved them to a provider serving the same model at a lower precision or a shorter context cap."}],"correct":true,"tempting":null},{"text":[{"text":"Aggregate demand raised latency, and slower generation produces lower-quality output."}],"correct":false,"tempting":[{"text":"Confuses speed with quality; the same weights and sampler produce the same distribution regardless of how long it takes."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"what-each-size-can-do","title":"What a model of each size can honestly do","minutes":9},"next":{"slug":"base-instruct-reasoning","title":"Base, Instruct, Reasoning","minutes":8}}}