{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"hosted-open-models","title":"Renting an open model instead of running it","url":"https://addaly.com/learn/choosing-your-tools/hosted-open-models","position":44,"of":76,"minutes":8,"answer":"Hosted open models sit between a laptop and a frontier subscription, and the same model name from two providers can differ in speed, quantisation, context length and data terms, so check those before comparing prices.","keyIdea":"Hosted open models sit between a laptop and a frontier subscription, and the same model name from two providers can differ in speed, quantisation, context length and data terms, so check those before comparing prices."},"body":{"format":"markdown","text":"## The route between a laptop and a frontier subscription\n\nSeveral companies do nothing but serve open models over an API: you send a request naming `llama-3.3-70b` or `qwen3-32b`, they run it on their hardware, you pay per token. Prices sit well below frontier models and the quality on ordinary tasks is close enough that the difference rarely shows.\n\nThis is the most under-used route in the whole field. It covers the large space between \"my laptop cannot run this\" and \"I am paying frontier prices for summarising\".\n\n## What you are comparing between providers\n\nThe same model name from two providers is not the same service. Five things differ and all of them are checkable before you commit.\n\n**Price per million tokens.** Compare input and output separately.\n\n**Speed.** Tokens per second varies enormously between providers running identical weights, because it depends on their hardware and serving stack. Some specialist providers are several times faster than others on the same model. If you are generating long documents, this is the difference between a workable tool and an irritating one.\n\n**Quantisation.** Some serve at full precision, some at 8-bit, some at 4-bit, and many do not say. A provider serving a heavily quantised version at a lower price is offering a different product under the same name. Ask, or look for a published statement; the good ones publish it.\n\n**Context length served.** Frequently lower than the model supports. A model advertised at 128k may be offered at 32k.\n\n**Data terms.** The important one. Some providers state plainly that they do not train on or retain your inputs; some have a free tier that does both; some are quiet about it. This is the question to settle before the price question, because a cheap provider that keeps your client's contract is not cheap.\n\n## Where this route wins\n\n**Bulk work.** Classification, extraction, summarising, translation, tagging: the tasks where open models match frontier models and volume makes price decisive.\n\n**Predictable, moderate use.** A few dollars a month rather than twenty.\n\n**Speed-critical work.** The fastest hosted open models generate several hundred tokens per second, which is faster than any frontier model and changes what interactive tools feel like.\n\n**Trying a model before installing it.** Cheaper than a 40 GB download you may not keep.\n\n**Data residency.** Some providers let you choose the region your requests are processed in, which matters where regulation requires it.\n\n## Where it does not\n\n**The hardest tasks.** Long multi-step reasoning, agentic tool use over many steps, and the deepest knowledge questions still favour frontier models. Test rather than assume; the gap has narrowed and it has not closed.\n\n**Confidentiality that must be structural.** A promise in a policy is not the same as computation on your own machine. If the answer must be \"the data never left the building\", this route is not it.\n\n**Feature parity.** No memory, no web search, no file store unless you build them.\n\n## Checking a provider in ten minutes\n\nBefore committing to one, run three things. Send the same prompt to two providers serving the same model and compare the answers side by side; large differences point at different precision. Time a 500-word generation on each and note the tokens per second. And read the data page rather than the marketing page, looking specifically for the words *retention*, *training* and *human review*.\n\nIf the third of those cannot be answered in ten minutes, that is your answer for anything sensitive. Nothing else about the provider needs checking, because the question that mattered has already been left unanswered.\n\n## A sensible arrangement\n\nFor most people who have got this far, the arrangement that works is three-way and costs very little:\n\n- **Local model** for anything confidential and anything offline.\n- **Hosted open model** for volume and for routine work, at a few dollars a month.\n- **One frontier model**, subscription or API, for the hard things.\n\nThe interesting part is how rarely the third is needed once the first two are set up properly. People who make this arrangement typically find that the frontier tool handles perhaps a tenth of their requests, and that the tenth is worth paying for while the other nine tenths were not."},"check":{"question":"Two providers serve the same open model at very different prices, and the cheaper one does not state its quantisation. What follows?","options":[{"text":"The price difference is most likely infrastructure efficiency, since quantisation choices are standardised across providers.","correct":false,"tempting":"Assumes a standard that does not exist; providers independently choose precision and many do not disclose it."},{"text":"The cheaper one is running at lower precision, and the released weights make that verifiable by comparing outputs.","correct":false,"tempting":"Jumps to a conclusion and offers a check that will not settle it, since sampling variation swamps quantisation differences in a few outputs."},{"text":"Quantisation is irrelevant to a hosted service, since the provider absorbs the memory cost rather than passing it on.","correct":false,"tempting":"Confuses who pays for memory with who feels the quality effect, which lands entirely on your answers."},{"text":"You may be buying a different product under the same name, so ask about precision and context before comparing on price.","correct":true}]},"prev":{"slug":"quantisation-in-practice","title":"Choosing a quantisation","url":"https://addaly.com/learn/choosing-your-tools/quantisation-in-practice"},"next":{"slug":"licences-as-a-decision","title":"The licence is a business decision","url":"https://addaly.com/learn/choosing-your-tools/licences-as-a-decision"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Renting an open model instead of running it', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/hosted-open-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Renting an open model instead of running it","courseTitle":"Choosing and Using the Tools","module":{"index":5,"slug":"open-models-and-your-own-hardware","title":"Open models and your own hardware"},"answer":null,"keyIdea":"Hosted open models sit between a laptop and a frontier subscription, and the same model name from two providers can differ in speed, quantisation, context length and data terms, so check those before comparing prices.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-route-between-a-laptop-and-a-frontier-subscription","runs":[{"text":"The route between a laptop and a frontier subscription"}]},{"t":"p","runs":[{"text":"Several companies do nothing but serve open models over an API: you send a request naming "},{"text":"llama-3.3-70b","code":true},{"text":" or "},{"text":"qwen3-32b","code":true},{"text":", they run it on their hardware, you pay per token. Prices sit well below frontier models and the quality on ordinary tasks is close enough that the difference rarely shows."}]},{"t":"p","runs":[{"text":"This is the most under-used route in the whole field. It covers the large space between “my laptop cannot run this” and “I am paying frontier prices for summarising”."}]},{"t":"h2","id":"what-you-are-comparing-between-providers","runs":[{"text":"What you are comparing between providers"}]},{"t":"p","runs":[{"text":"The same model name from two providers is not the same service. Five things differ and all of them are checkable before you commit."}]},{"t":"p","runs":[{"text":"Price per million tokens.","b":true},{"text":" Compare input and output separately."}]},{"t":"p","runs":[{"text":"Speed.","b":true},{"text":" Tokens per second varies enormously between providers running identical weights, because it depends on their hardware and serving stack. Some specialist providers are several times faster than others on the same model. If you are generating long documents, this is the difference between a workable tool and an irritating one."}]},{"t":"p","runs":[{"text":"Quantisation.","b":true},{"text":" Some serve at full precision, some at 8-bit, some at 4-bit, and many do not say. A provider serving a heavily quantised version at a lower price is offering a different product under the same name. Ask, or look for a published statement; the good ones publish it."}]},{"t":"p","runs":[{"text":"Context length served.","b":true},{"text":" Frequently lower than the model supports. A model advertised at 128k may be offered at 32k."}]},{"t":"p","runs":[{"text":"Data terms.","b":true},{"text":" The important one. Some providers state plainly that they do not train on or retain your inputs; some have a free tier that does both; some are quiet about it. This is the question to settle before the price question, because a cheap provider that keeps your client’s contract is not cheap."}]},{"t":"h2","id":"where-this-route-wins","runs":[{"text":"Where this route wins"}]},{"t":"p","runs":[{"text":"Bulk work.","b":true},{"text":" Classification, extraction, summarising, translation, tagging: the tasks where open models match frontier models and volume makes price decisive."}]},{"t":"p","runs":[{"text":"Predictable, moderate use.","b":true},{"text":" A few dollars a month rather than twenty."}]},{"t":"p","runs":[{"text":"Speed-critical work.","b":true},{"text":" The fastest hosted open models generate several hundred tokens per second, which is faster than any frontier model and changes what interactive tools feel like."}]},{"t":"p","runs":[{"text":"Trying a model before installing it.","b":true},{"text":" Cheaper than a 40 GB download you may not keep."}]},{"t":"p","runs":[{"text":"Data residency.","b":true},{"text":" Some providers let you choose the region your requests are processed in, which matters where regulation requires it."}]},{"t":"h2","id":"where-it-does-not","runs":[{"text":"Where it does not"}]},{"t":"p","runs":[{"text":"The hardest tasks.","b":true},{"text":" Long multi-step reasoning, agentic tool use over many steps, and the deepest knowledge questions still favour frontier models. Test rather than assume; the gap has narrowed and it has not closed."}]},{"t":"p","runs":[{"text":"Confidentiality that must be structural.","b":true},{"text":" A promise in a policy is not the same as computation on your own machine. If the answer must be “the data never left the building”, this route is not it."}]},{"t":"p","runs":[{"text":"Feature parity.","b":true},{"text":" No memory, no web search, no file store unless you build them."}]},{"t":"h2","id":"checking-a-provider-in-ten-minutes","runs":[{"text":"Checking a provider in ten minutes"}]},{"t":"p","runs":[{"text":"Before committing to one, run three things. Send the same prompt to two providers serving the same model and compare the answers side by side; large differences point at different precision. Time a 500-word generation on each and note the tokens per second. And read the data page rather than the marketing page, looking specifically for the words "},{"text":"retention","i":true},{"text":", "},{"text":"training","i":true},{"text":" and "},{"text":"human review","i":true},{"text":"."}]},{"t":"p","runs":[{"text":"If the third of those cannot be answered in ten minutes, that is your answer for anything sensitive. Nothing else about the provider needs checking, because the question that mattered has already been left unanswered."}]},{"t":"h2","id":"a-sensible-arrangement","runs":[{"text":"A sensible arrangement"}]},{"t":"p","runs":[{"text":"For most people who have got this far, the arrangement that works is three-way and costs very little:"}]},{"t":"ul","items":[[{"text":"Local model","b":true},{"text":" for anything confidential and anything offline."}],[{"text":"Hosted open model","b":true},{"text":" for volume and for routine work, at a few dollars a month."}],[{"text":"One frontier model","b":true},{"text":", subscription or API, for the hard things."}]]},{"t":"p","runs":[{"text":"The interesting part is how rarely the third is needed once the first two are set up properly. People who make this arrangement typically find that the frontier tool handles perhaps a tenth of their requests, and that the tenth is worth paying for while the other nine tenths were not."}]}],"check":{"question":[{"text":"Two providers serve the same open model at very different prices, and the cheaper one does not state its quantisation. What follows?"}],"options":[{"text":[{"text":"The price difference is most likely infrastructure efficiency, since quantisation choices are standardised across providers."}],"correct":false,"tempting":[{"text":"Assumes a standard that does not exist; providers independently choose precision and many do not disclose it."}]},{"text":[{"text":"The cheaper one is running at lower precision, and the released weights make that verifiable by comparing outputs."}],"correct":false,"tempting":[{"text":"Jumps to a conclusion and offers a check that will not settle it, since sampling variation swamps quantisation differences in a few outputs."}]},{"text":[{"text":"Quantisation is irrelevant to a hosted service, since the provider absorbs the memory cost rather than passing it on."}],"correct":false,"tempting":[{"text":"Confuses who pays for memory with who feels the quality effect, which lands entirely on your answers."}]},{"text":[{"text":"You may be buying a different product under the same name, so ask about precision and context before comparing on price."}],"correct":true,"tempting":null}],"order":[1,0,3,2],"why":null},"prev":{"slug":"quantisation-in-practice","title":"Choosing a quantisation","minutes":8},"next":{"slug":"licences-as-a-decision","title":"The licence is a business decision","minutes":8}}}