Serving it to other people

Once someone else depends on your endpoint, three things change: the API surface, the concurrency behaviour, and the fact that a machine on your network now takes instructions from strangers.

The endpoint

Every serious engine speaks the OpenAI dialect, so clients are interchangeable.

vllm serve Qwen/Qwen3-8B-AWQ \
  --host 127.0.0.1 --port 8000 \
  --api-key "$LOCAL_API_KEY" \
  --max-model-len 16384 \
  --enable-prefix-caching

llama-server -m model.gguf \
  -c 32768 --parallel 4 \
  --api-key-file /etc/llama/key \
  --host 127.0.0.1 --port 8080

A trap in that second command. In llama-server, -c is the *total* KV budget shared across slots. --parallel 4 -c 32768 gives each of four concurrent users 8,192 tokens, not 32,768. People meet this as mysterious truncation that only happens when the team is busy.

Concurrency

Two numbers matter, and neither is average tokens per second: p95 time to first token (what makes it feel slow) and aggregate tokens per second (what sets your cost).

Two features do most of the work.

Continuous batching admits new requests into the running batch instead of waiting for the current one to finish. Without it, one long generation blocks everyone behind it.

Chunked prefill splits a large prompt into pieces and interleaves them with ongoing decoding, so one person pasting a 30,000-token document does not freeze everyone else's chat mid-sentence. In vLLM this is on by default in recent versions.

Prefix caching, which is usually the biggest win

If thirty requests share a 2,000-token system prompt, that prefix is being recomputed thirty times. Cache it once and you delete most of your prefill work.

• vLLM: --enable-prefix-caching
• SGLang: RadixAttention, on by default, and it matches partial prefixes automatically
• llama.cpp: slot reuse, which keeps a conversation on the same slot so its cache survives

For agent workloads with a long fixed preamble, this is worth more than changing engines.

Security

The default posture is: bind to 127.0.0.1. Then pick one of two ways out.

1. 1A private network. Tailscale or WireGuard. Your users join the network; the port is never on the public internet. This is the simplest correct answer for a team, and it is what most people should do.
2. 2A reverse proxy — Caddy or nginx — terminating TLS and enforcing authentication, with the engine itself still on localhost. Never publish the engine port directly.

Why this matters more than it sounds:

• An open endpoint is a free compute account for anyone who finds it, and they do find it. Port scanners index the default inference ports continuously; there have been repeated waves of exposed Ollama instances discovered this way.
• It is a data leak. Many servers log prompts by default.
• It is trivially denied. One request with a maximum-length context ties up the KV cache.
• Some management endpoints do more than inference. An exposed Ollama instance exposes model pull and delete: a stranger can fill your disk or remove your models.

Also be clear about what the built-in API key is. It is one shared bearer token, with no per-user accounting and no rate limiting. If you need those, put a gateway in front — LiteLLM's proxy, or Open WebUI's own account system — and keep the engine private behind it.

Finally: the model is not a security boundary. Anyone who can send prompts can extract your system prompt. If you have given the model tools, everyone chatting to it has those tools. A model with shell access is a shell.

Before you invite anyone

Run a load test at the concurrency you expect, with prompts the length yours will be, and watch p95 time to first token. "It felt fast when I tried it" is a measurement of one user on an idle machine, which is the one condition that will never occur again.