{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"inference-providers","title":"Renting the compute instead of owning it","url":"https://addaly.com/learn/hugging-face/inference-providers","position":70,"of":75,"minutes":8,"answer":"The Hub routes inference to third-party providers behind one OpenAI-compatible interface, and the choice between serverless, a dedicated endpoint, your own server and the user's device turns on token volume against GPU utilisation.","keyIdea":"The Hub routes inference to third-party providers behind one OpenAI-compatible interface, and the choice between serverless, a dedicated endpoint, your own server and the user's device turns on token volume against GPU utilisation, with the crossover much higher than teams assume and privacy deciding it outright for sensitive data."},"body":{"format":"markdown","text":"## The Hub as a router\n\nMany model pages carry an inference widget and a code snippet that calls a hosted API. Behind it sits a router: Hugging Face forwards your request to one of several third-party providers (Together, Fireworks, Replicate, SambaNova, Cerebras and others) under a single token and a single bill.\n\n```python\nfrom huggingface_hub import InferenceClient\n\nclient = InferenceClient(api_key=os.environ[\"HF_TOKEN\"])\n\nout = client.chat_completion(\n    model=\"Qwen/Qwen2.5-7B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarise this in two lines: ...\"}],\n    max_tokens=200,\n)\nprint(out.choices[0].message.content)\n```\n\nThe interface is OpenAI-compatible, so the `openai` package pointed at the router's base URL works unchanged, which matters, because it means switching between a hosted model, a local llama.cpp server and a provider is a change of base URL rather than a rewrite.\n\nThere is a small monthly credit on free accounts and a larger one on paid; beyond that you are billed at the provider's rates with no markup, which is stated on the pricing page and worth re-reading, since these arrangements change.\n\n> Figure: Cost a day: serverless against one rented card. Line chart of US dollars a day against millions of tokens a day: Serverless, billed per token (0, 0) (50, 15) (100, 30) (150, 45) (200, 60); One rented GPU, kept busy (0, 24) (50, 24) (100, 24) (150, 24) (200, 48). Order-of-magnitude figures rather than quotes, and the rates move. The shape is what transfers: per-token billing is a line through the origin, a rented card is flat until you need a second one, and the crossover sits much further right than teams assume. It moves right again by however idle the card is, because idle costs the same as busy.\n\n## The four options, and the honest comparison\n\n**Serverless through a provider.** No infrastructure, per-token billing, cold starts on less popular models, and your input goes to a third party. Right for prototypes, spiky traffic and anything where you would rather not run a GPU.\n\n**A dedicated endpoint.** Hugging Face Inference Endpoints, or the equivalent on a cloud, gives you a container running your model on hardware you rent by the hour, with autoscaling and scale-to-zero available. Predictable latency, private, and billed for uptime rather than requests: the same trap as paid Spaces, with the same fix: configure scale-to-zero before you configure anything else.\n\n**Your own server.** vLLM or text-generation-inference on a rented or owned GPU. Cheapest per token at volume, most work, complete control of data.\n\n**On the user's device.** transformers.js in a browser, GGUF on a laptop, a small model on a phone. Zero marginal cost, complete privacy, limited to small models. Module three covered it, and it remains the most underused option.\n\n## The arithmetic that decides it\n\nThe crossover is not where people guess. Take a rough figure of a few tens of US cents per million tokens for a small open model through a provider, and around a dollar an hour for a modest GPU.\n\nAt 100,000 tokens a day, serverless costs a few cents a day and a dedicated GPU costs twenty-four dollars. Serverless wins by a very wide margin.\n\nAt 100 million tokens a day, serverless is in the tens of dollars a day and one busy GPU is still twenty-four. Now owning wins, provided you can keep it busy, and utilisation is the variable nobody models. A GPU idle 80% of the time costs the same as a GPU at full load, so your effective rate is five times the headline one.\n\n**Run the arithmetic with your own volume before choosing.** Most teams self-host too early, and a smaller number stay on per-token billing long past the point where a single card would be cheaper.\n\n## Privacy is not a footnote here\n\nSending data to a provider means it leaves your machine. What is logged, for how long, and whether it may be used for training differs by provider and by plan, and the answer lives in their terms rather than in the Hub's interface. For health, financial, legal or children's data (or anything a user would not have pasted into a public website) that question has to be answered before the integration is written, not after.\n\nThe local path is available and free. A 3B model at `Q4_K_M` on a CPU handles a great deal of routine work, and nothing leaves the building.\n\n## Reliability, stated plainly\n\nA provider can deprecate a model, change its quantization, rate limit you, or go down. All four have happened to somebody this year. Three mitigations cost little:\n\n1. **Keep the provider behind one function** in your code, so swapping is a small change.\n2. **Have a fallback**, even a smaller local model, for when the primary is unavailable.\n3. **Cache aggressively.** A large share of production requests are repeats, and a hash-keyed cache is often the single biggest cost saving available.\n\n## Do this now\n\nEstimate your monthly token volume, then compute what it costs serverless and what one rented GPU costs for the same month. Write both numbers down. Most people have never done this and are surprised by which side they are on."},"check":{"question":"A service handles about 200,000 tokens a day and moves from per-token billing to a dedicated GPU endpoint to save money. The bill rises. Why?","options":[{"text":"Dedicated endpoints add a management fee on top of the hourly hardware rate, which per-token pricing avoids.","correct":false,"tempting":"Focuses on a pricing component rather than the structural difference between paying for time and paying for use."},{"text":"Endpoint hardware is billed per hour whether or not requests arrive, and this volume leaves it mostly idle.","correct":true},{"text":"Cold starts on a scale-to-zero endpoint trigger repeated billed initialisations that exceed the steady-state cost.","correct":false,"tempting":"Scale-to-zero reduces the bill in this scenario, and initialisation is not separately metered."},{"text":"Dedicated endpoints cannot batch requests, so each one occupies the GPU exclusively and throughput falls.","correct":false,"tempting":"Serving stacks on endpoints batch continuously, and throughput is not the binding cost factor at this volume."}]},"prev":{"slug":"thirty-examples-of-your-own","title":"The evaluation that decides it","url":"https://addaly.com/learn/hugging-face/thirty-examples-of-your-own"},"next":{"slug":"serving-it-yourself","title":"When transformers stops being the right server","url":"https://addaly.com/learn/hugging-face/serving-it-yourself"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Renting the compute instead of owning it', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/inference-providers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Renting the compute instead of owning it","courseTitle":"Hugging Face, End to End","module":{"index":8,"slug":"depending-on-it-and-publishing","title":"Depending on it, and publishing so others can"},"answer":null,"keyIdea":"The Hub routes inference to third-party providers behind one OpenAI-compatible interface, and the choice between serverless, a dedicated endpoint, your own server and the user’s device turns on token volume against GPU utilisation, with the crossover much higher than teams assume and privacy deciding it outright for sensitive data.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Cost a day: serverless against one rented card","series":[{"label":"Serverless, billed per token","points":[[0,0],[50,15],[100,30],[150,45],[200,60]]},{"label":"One rented GPU, kept busy","points":[[0,24],[50,24],[100,24],[150,24],[200,48]]}],"xLabel":"millions of tokens a day","yLabel":"US dollars a day","caption":"Order-of-magnitude figures rather than quotes, and the rates move. The shape is what transfers: per-token billing is a line through the origin, a rented card is flat until you need a second one, and the crossover sits much further right than teams assume. It moves right again by however idle the card is, because idle costs the same as busy."},"alt":"Figure: Cost a day: serverless against one rented card. Line chart of US dollars a day against millions of tokens a day: Serverless, billed per token (0, 0) (50, 15) (100, 30) (150, 45) (200, 60); One rented GPU, kept busy (0, 24) (50, 24) (100, 24) (150, 24) (200, 48). Order-of-magnitude figures rather than quotes, and the rates move. The shape is what transfers: per-token billing is a line through the origin, a rented card is flat until you need a second one, and the crossover sits much further right than teams assume. It moves right again by however idle the card is, because idle costs the same as busy."}},"blocks":[{"t":"h2","id":"the-hub-as-a-router","runs":[{"text":"The Hub as a router"}]},{"t":"p","runs":[{"text":"Many model pages carry an inference widget and a code snippet that calls a hosted API. Behind it sits a router: Hugging Face forwards your request to one of several third-party providers (Together, Fireworks, Replicate, SambaNova, Cerebras and others) under a single token and a single bill."}]},{"t":"code","lang":"python","label":"Python","text":"from huggingface_hub import InferenceClient\n\nclient = InferenceClient(api_key=os.environ[\"HF_TOKEN\"])\n\nout = client.chat_completion(\n    model=\"Qwen/Qwen2.5-7B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarise this in two lines: ...\"}],\n    max_tokens=200,\n)\nprint(out.choices[0].message.content)"},{"t":"p","runs":[{"text":"The interface is OpenAI-compatible, so the "},{"text":"openai","code":true},{"text":" package pointed at the router’s base URL works unchanged, which matters, because it means switching between a hosted model, a local llama.cpp server and a provider is a change of base URL rather than a rewrite."}]},{"t":"p","runs":[{"text":"There is a small monthly credit on free accounts and a larger one on paid; beyond that you are billed at the provider’s rates with no markup, which is stated on the pricing page and worth re-reading, since these arrangements change."}]},{"t":"h2","id":"the-four-options-and-the-honest-comparison","runs":[{"text":"The four options, and the honest comparison"}]},{"t":"p","runs":[{"text":"Serverless through a provider.","b":true},{"text":" No infrastructure, per-token billing, cold starts on less popular models, and your input goes to a third party. Right for prototypes, spiky traffic and anything where you would rather not run a GPU."}]},{"t":"p","runs":[{"text":"A dedicated endpoint.","b":true},{"text":" Hugging Face Inference Endpoints, or the equivalent on a cloud, gives you a container running your model on hardware you rent by the hour, with autoscaling and scale-to-zero available. Predictable latency, private, and billed for uptime rather than requests: the same trap as paid Spaces, with the same fix: configure scale-to-zero before you configure anything else."}]},{"t":"p","runs":[{"text":"Your own server.","b":true},{"text":" vLLM or text-generation-inference on a rented or owned GPU. Cheapest per token at volume, most work, complete control of data."}]},{"t":"p","runs":[{"text":"On the user’s device.","b":true},{"text":" transformers.js in a browser, GGUF on a laptop, a small model on a phone. Zero marginal cost, complete privacy, limited to small models. Module three covered it, and it remains the most underused option."}]},{"t":"h2","id":"the-arithmetic-that-decides-it","runs":[{"text":"The arithmetic that decides it"}]},{"t":"p","runs":[{"text":"The crossover is not where people guess. Take a rough figure of a few tens of US cents per million tokens for a small open model through a provider, and around a dollar an hour for a modest GPU."}]},{"t":"p","runs":[{"text":"At 100,000 tokens a day, serverless costs a few cents a day and a dedicated GPU costs twenty-four dollars. Serverless wins by a very wide margin."}]},{"t":"p","runs":[{"text":"At 100 million tokens a day, serverless is in the tens of dollars a day and one busy GPU is still twenty-four. Now owning wins, provided you can keep it busy, and utilisation is the variable nobody models. A GPU idle 80% of the time costs the same as a GPU at full load, so your effective rate is five times the headline one."}]},{"t":"p","runs":[{"text":"Run the arithmetic with your own volume before choosing.","b":true},{"text":" Most teams self-host too early, and a smaller number stay on per-token billing long past the point where a single card would be cheaper."}]},{"t":"h2","id":"privacy-is-not-a-footnote-here","runs":[{"text":"Privacy is not a footnote here"}]},{"t":"p","runs":[{"text":"Sending data to a provider means it leaves your machine. What is logged, for how long, and whether it may be used for training differs by provider and by plan, and the answer lives in their terms rather than in the Hub’s interface. For health, financial, legal or children’s data (or anything a user would not have pasted into a public website) that question has to be answered before the integration is written, not after."}]},{"t":"p","runs":[{"text":"The local path is available and free. A 3B model at "},{"text":"Q4_K_M","code":true},{"text":" on a CPU handles a great deal of routine work, and nothing leaves the building."}]},{"t":"h2","id":"reliability-stated-plainly","runs":[{"text":"Reliability, stated plainly"}]},{"t":"p","runs":[{"text":"A provider can deprecate a model, change its quantization, rate limit you, or go down. All four have happened to somebody this year. Three mitigations cost little:"}]},{"t":"ol","start":1,"items":[[{"text":"Keep the provider behind one function","b":true},{"text":" in your code, so swapping is a small change."}],[{"text":"Have a fallback","b":true},{"text":", even a smaller local model, for when the primary is unavailable."}],[{"text":"Cache aggressively.","b":true},{"text":" A large share of production requests are repeats, and a hash-keyed cache is often the single biggest cost saving available."}]]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Estimate your monthly token volume, then compute what it costs serverless and what one rented GPU costs for the same month. Write both numbers down. Most people have never done this and are surprised by which side they are on."}]}],"check":{"question":[{"text":"A service handles about 200,000 tokens a day and moves from per-token billing to a dedicated GPU endpoint to save money. The bill rises. Why?"}],"options":[{"text":[{"text":"Dedicated endpoints add a management fee on top of the hourly hardware rate, which per-token pricing avoids."}],"correct":false,"tempting":[{"text":"Focuses on a pricing component rather than the structural difference between paying for time and paying for use."}]},{"text":[{"text":"Endpoint hardware is billed per hour whether or not requests arrive, and this volume leaves it mostly idle."}],"correct":true,"tempting":null},{"text":[{"text":"Cold starts on a scale-to-zero endpoint trigger repeated billed initialisations that exceed the steady-state cost."}],"correct":false,"tempting":[{"text":"Scale-to-zero reduces the bill in this scenario, and initialisation is not separately metered."}]},{"text":[{"text":"Dedicated endpoints cannot batch requests, so each one occupies the GPU exclusively and throughput falls."}],"correct":false,"tempting":[{"text":"Serving stacks on endpoints batch continuously, and throughput is not the binding cost factor at this volume."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"thirty-examples-of-your-own","title":"The evaluation that decides it","minutes":9},"next":{"slug":"serving-it-yourself","title":"When transformers stops being the right server","minutes":8}}}