{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"serving-it-to-others","title":"Serving it to other people","url":"https://addaly.com/learn/running-models-yourself/serving-it-to-others","position":56,"of":82,"minutes":9,"answer":"Bind to localhost and reach it over a private network; an open inference port is free compute for strangers.","keyIdea":"Bind to localhost and reach it over a private network; an open inference port is free compute for strangers."},"body":{"format":"markdown","text":"Once someone else depends on your endpoint, three things change: the API surface, the concurrency behaviour, and the fact that a machine on your network now takes instructions from strangers.\n\n## The endpoint\n\nEvery serious engine speaks the OpenAI dialect, so clients are interchangeable.\n\n```bash\nvllm serve Qwen/Qwen3-8B-AWQ \\\n  --host 127.0.0.1 --port 8000 \\\n  --api-key \"$LOCAL_API_KEY\" \\\n  --max-model-len 16384 \\\n  --enable-prefix-caching\n```\n\n```bash\nllama-server -m model.gguf \\\n  -c 32768 --parallel 4 \\\n  --api-key-file /etc/llama/key \\\n  --host 127.0.0.1 --port 8080\n```\n\n**A trap in that second command.** In `llama-server`, `-c` is the *total* KV budget shared across slots. `--parallel 4 -c 32768` gives each of four concurrent users 8,192 tokens, not 32,768. People meet this as mysterious truncation that only happens when the team is busy.\n\n## Concurrency\n\nTwo numbers matter, and neither is average tokens per second: **p95 time to first token** (what makes it feel slow) and **aggregate tokens per second** (what sets your cost).\n\nTwo features do most of the work.\n\n**Continuous batching** admits new requests into the running batch instead of waiting for the current one to finish. Without it, one long generation blocks everyone behind it.\n\n**Chunked prefill** splits a large prompt into pieces and interleaves them with ongoing decoding, so one person pasting a 30,000-token document does not freeze everyone else's chat mid-sentence. In vLLM this is on by default in recent versions.\n\n## Prefix caching, which is usually the biggest win\n\nIf thirty requests share a 2,000-token system prompt, that prefix is being recomputed thirty times. Cache it once and you delete most of your prefill work.\n\n- vLLM: `--enable-prefix-caching`\n- SGLang: RadixAttention, on by default, and it matches partial prefixes automatically\n- llama.cpp: slot reuse, which keeps a conversation on the same slot so its cache survives\n\nFor agent workloads with a long fixed preamble, this is worth more than changing engines.\n\n## Security\n\nThe default posture is: **bind to 127.0.0.1**. Then pick one of two ways out.\n\n1. **A private network.** Tailscale or WireGuard. Your users join the network; the port is never on the public internet. This is the simplest correct answer for a team, and it is what most people should do.\n2. **A reverse proxy** (Caddy or nginx) terminating TLS and enforcing authentication, with the engine itself still on localhost. Never publish the engine port directly.\n\nWhy this matters more than it sounds:\n\n- **An open endpoint is a free compute account for anyone who finds it,** and they do find it. Port scanners index the default inference ports continuously; there have been repeated waves of exposed Ollama instances discovered this way.\n- **It is a data leak.** Many servers log prompts by default.\n- **It is trivially denied.** One request with a maximum-length context ties up the KV cache.\n- **Some management endpoints do more than inference.** An exposed Ollama instance exposes model pull and delete: a stranger can fill your disk or remove your models.\n\nAlso be clear about what the built-in API key is. It is one shared bearer token, with no per-user accounting and no rate limiting. If you need those, put a gateway in front (LiteLLM's proxy, or Open WebUI's own account system) and keep the engine private behind it.\n\nFinally: **the model is not a security boundary.** Anyone who can send prompts can extract your system prompt. If you have given the model tools, everyone chatting to it has those tools. A model with shell access is a shell.\n\n## Before you invite anyone\n\nRun a load test at the concurrency you expect, with prompts the length yours will be, and watch p95 time to first token. \"It felt fast when I tried it\" is a measurement of one user on an idle machine, which is the one condition that will never occur again."},"check":{"question":"You run `llama-server -c 32768 --parallel 4` for four teammates. Each reports the model forgetting earlier messages after roughly 8,000 tokens, even though the model supports far more. What is happening?","options":[{"text":"`-c` sets the total KV budget shared across the four slots, so each user gets 8,192 tokens; raise `-c`, quantise the cache, or reduce `--parallel`.","correct":true},{"text":"The model's genuine context limit is around 8k despite the claim on its card.","correct":false,"tempting":"Attributes a server allocation decision to the weights, which is checkable and wrong here."},{"text":"The four users share one cache, so their conversations overwrite each other once it fills.","correct":false,"tempting":"Slots are isolated; if they were not, users would see fragments of each other's chats rather than clean forgetting."},{"text":"`--parallel` is per connection, so four users need `--parallel 16` to get the full window each.","correct":false,"tempting":"Misreads the flag's direction; raising it would divide the same budget further and make the problem worse."}]},"prev":{"slug":"a-local-eval-harness","title":"A small harness that tells you whether a change helped","url":"https://addaly.com/learn/running-models-yourself/a-local-eval-harness"},"next":{"slug":"reaching-the-machine-safely","title":"Reaching the machine from somewhere else","url":"https://addaly.com/learn/running-models-yourself/reaching-the-machine-safely"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Serving it to other people', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/serving-it-to-others (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Serving it to other people","courseTitle":"Running Models Yourself","module":{"index":7,"slug":"serving-it-to-others","title":"Serving it to other people"},"answer":null,"keyIdea":"Bind to localhost and reach it over a private network; an open inference port is free compute for strangers.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"p","runs":[{"text":"Once someone else depends on your endpoint, three things change: the API surface, the concurrency behaviour, and the fact that a machine on your network now takes instructions from strangers."}],"lede":true},{"t":"h2","id":"the-endpoint","runs":[{"text":"The endpoint"}]},{"t":"p","runs":[{"text":"Every serious engine speaks the OpenAI dialect, so clients are interchangeable."}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve Qwen/Qwen3-8B-AWQ \\\n  --host 127.0.0.1 --port 8000 \\\n  --api-key \"$LOCAL_API_KEY\" \\\n  --max-model-len 16384 \\\n  --enable-prefix-caching"},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf \\\n  -c 32768 --parallel 4 \\\n  --api-key-file /etc/llama/key \\\n  --host 127.0.0.1 --port 8080"},{"t":"p","runs":[{"text":"A trap in that second command.","b":true},{"text":" In "},{"text":"llama-server","code":true},{"text":", "},{"text":"-c","code":true},{"text":" is the "},{"text":"total","i":true},{"text":" KV budget shared across slots. "},{"text":"--parallel 4 -c 32768","code":true},{"text":" gives each of four concurrent users 8,192 tokens, not 32,768. People meet this as mysterious truncation that only happens when the team is busy."}]},{"t":"h2","id":"concurrency","runs":[{"text":"Concurrency"}]},{"t":"p","runs":[{"text":"Two numbers matter, and neither is average tokens per second: "},{"text":"p95 time to first token","b":true},{"text":" (what makes it feel slow) and "},{"text":"aggregate tokens per second","b":true},{"text":" (what sets your cost)."}]},{"t":"p","runs":[{"text":"Two features do most of the work."}]},{"t":"p","runs":[{"text":"Continuous batching","b":true},{"text":" admits new requests into the running batch instead of waiting for the current one to finish. Without it, one long generation blocks everyone behind it."}]},{"t":"p","runs":[{"text":"Chunked prefill","b":true},{"text":" splits a large prompt into pieces and interleaves them with ongoing decoding, so one person pasting a 30,000-token document does not freeze everyone else’s chat mid-sentence. In vLLM this is on by default in recent versions."}]},{"t":"h2","id":"prefix-caching-which-is-usually-the-biggest-win","runs":[{"text":"Prefix caching, which is usually the biggest win"}]},{"t":"p","runs":[{"text":"If thirty requests share a 2,000-token system prompt, that prefix is being recomputed thirty times. Cache it once and you delete most of your prefill work."}]},{"t":"ul","items":[[{"text":"vLLM: "},{"text":"--enable-prefix-caching","code":true}],[{"text":"SGLang: RadixAttention, on by default, and it matches partial prefixes automatically"}],[{"text":"llama.cpp: slot reuse, which keeps a conversation on the same slot so its cache survives"}]]},{"t":"p","runs":[{"text":"For agent workloads with a long fixed preamble, this is worth more than changing engines."}]},{"t":"h2","id":"security","runs":[{"text":"Security"}]},{"t":"p","runs":[{"text":"The default posture is: "},{"text":"bind to 127.0.0.1","b":true},{"text":". Then pick one of two ways out."}]},{"t":"ol","start":1,"items":[[{"text":"A private network.","b":true},{"text":" Tailscale or WireGuard. Your users join the network; the port is never on the public internet. This is the simplest correct answer for a team, and it is what most people should do."}],[{"text":"A reverse proxy","b":true},{"text":" (Caddy or nginx) terminating TLS and enforcing authentication, with the engine itself still on localhost. Never publish the engine port directly."}]]},{"t":"p","runs":[{"text":"Why this matters more than it sounds:"}]},{"t":"ul","items":[[{"text":"An open endpoint is a free compute account for anyone who finds it,","b":true},{"text":" and they do find it. Port scanners index the default inference ports continuously; there have been repeated waves of exposed Ollama instances discovered this way."}],[{"text":"It is a data leak.","b":true},{"text":" Many servers log prompts by default."}],[{"text":"It is trivially denied.","b":true},{"text":" One request with a maximum-length context ties up the KV cache."}],[{"text":"Some management endpoints do more than inference.","b":true},{"text":" An exposed Ollama instance exposes model pull and delete: a stranger can fill your disk or remove your models."}]]},{"t":"p","runs":[{"text":"Also be clear about what the built-in API key is. It is one shared bearer token, with no per-user accounting and no rate limiting. If you need those, put a gateway in front (LiteLLM’s proxy, or Open WebUI’s own account system) and keep the engine private behind it."}]},{"t":"p","runs":[{"text":"Finally: "},{"text":"the model is not a security boundary.","b":true},{"text":" Anyone who can send prompts can extract your system prompt. If you have given the model tools, everyone chatting to it has those tools. A model with shell access is a shell."}]},{"t":"h2","id":"before-you-invite-anyone","runs":[{"text":"Before you invite anyone"}]},{"t":"p","runs":[{"text":"Run a load test at the concurrency you expect, with prompts the length yours will be, and watch p95 time to first token. “It felt fast when I tried it” is a measurement of one user on an idle machine, which is the one condition that will never occur again."}]}],"check":{"question":[{"text":"You run "},{"text":"llama-server -c 32768 --parallel 4","code":true},{"text":" for four teammates. Each reports the model forgetting earlier messages after roughly 8,000 tokens, even though the model supports far more. What is happening?"}],"options":[{"text":[{"text":"-c","code":true},{"text":" sets the total KV budget shared across the four slots, so each user gets 8,192 tokens; raise "},{"text":"-c","code":true},{"text":", quantise the cache, or reduce "},{"text":"--parallel","code":true},{"text":"."}],"correct":true,"tempting":null},{"text":[{"text":"The model’s genuine context limit is around 8k despite the claim on its card."}],"correct":false,"tempting":[{"text":"Attributes a server allocation decision to the weights, which is checkable and wrong here."}]},{"text":[{"text":"The four users share one cache, so their conversations overwrite each other once it fills."}],"correct":false,"tempting":[{"text":"Slots are isolated; if they were not, users would see fragments of each other’s chats rather than clean forgetting."}]},{"text":[{"text":"--parallel","code":true},{"text":" is per connection, so four users need "},{"text":"--parallel 16","code":true},{"text":" to get the full window each."}],"correct":false,"tempting":[{"text":"Misreads the flag’s direction; raising it would divide the same budget further and make the problem worse."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"a-local-eval-harness","title":"A small harness that tells you whether a change helped","minutes":9},"next":{"slug":"reaching-the-machine-safely","title":"Reaching the machine from somewhere else","minutes":9}}}