A gateway in front: keys, quotas and a bill

What the engine deliberately does not do

vLLM, llama.cpp’s server and Ollama all implement inference and almost nothing else. vLLM has a single --api-key shared secret, which is better than nothing and worse than useless the moment more than one person holds it: you cannot tell who spent the GPU hour, and revoking it locks out everybody. Ollama has no authentication at all. This is not an oversight. Identity, quotas, billing and audit belong in a layer that outlives whichever engine you are running this month.

That layer is a gateway: a small service that sits between callers and the engine, holds the keys, counts the tokens and forwards the request. Because every engine speaks the same OpenAI-shaped API, the gateway is engine-agnostic.

The free option that does the most

LiteLLM Proxy is open source under MIT and runs as one container. It gives you virtual keys per person or per application, budgets in currency and in tokens, per-key rate limits, request logs, and a model-name mapping so callers ask for internal-fast while you decide whether that is a local Qwen or a hosted model this week.

# config.yaml
model_list:
  - model_name: internal-fast
    litellm_params:
      model: openai/qwen2.5-7b-instruct
      api_base: http://127.0.0.1:8000/v1
      api_key: dummy
litellm_settings:
  max_budget: 40          # currency units per month, all keys
  budget_duration: 30d

docker run -p 4000:4000 -v $(pwd)/config.yaml:/app/config.yaml \
  -e LITELLM_MASTER_KEY=sk-... ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

# then mint a key with its own ceiling
curl -X POST http://127.0.0.1:4000/key/generate \
  -H 'Authorization: Bearer sk-...' -H 'Content-Type: application/json' \
  -d '{"models":["internal-fast"],"max_budget":5,"rpm_limit":20,"metadata":{"user":"priya"}}'

The key that comes back is the only thing a caller ever sees. Delete it and that person is out, and nobody else notices.

The smaller alternative is a reverse proxy you already understand. Caddy in about six lines will terminate TLS with an automatically issued certificate and check a bearer token; nginx does the same with auth_request or a map of tokens. You get authentication and TLS but no accounting, which is fine when the whole audience is three colleagues.

Rate limits are three different limits

People write one number and are surprised when the machine still falls over.

• Requests per minute protects against a loop in somebody’s script. Cheap to enforce, and it does not protect the GPU, because one request can be enormous.
• Tokens per minute protects the GPU. This is the limit that maps to the hardware, because throughput is tokens, not calls.
• Concurrency (how many of one caller’s requests may be in flight at once) protects everybody else’s latency. Without it, one batch job with 200 parallel workers fills the engine’s queue and every interactive user waits behind it.

Set all three. A reasonable starting point for an internal service on one card: 60 requests per minute, tokens per minute at roughly half your measured sustained throughput, and a concurrency of four per key.

Set the maximum output length at the gateway

The single most effective protection is a cap the caller cannot raise. A request with no max_tokens and a model that will not stop occupies a slot until it hits the context limit: on a 32k context that is minutes of exclusive GPU time for one caller. Enforce a default and a ceiling in the gateway, and let callers request less but never more.

Most “the server hung” reports are one runaway generation holding a slot while everything else queues behind it.

What the gateway should refuse to pass through

Engines expose management endpoints that have no business being reachable by callers. Ollama’s API can pull a model from the internet onto your disk (/api/pull), delete one (/api/delete) and create new ones (/api/create). vLLM can expose a metrics endpoint and, if enabled, adapter loading. Allow-list the paths the gateway forwards (chat completions, completions, embeddings, models) and drop everything else. This is one line of configuration and it removes a whole category of problem.

Keep the engine on loopback

With a gateway in place the engine binds to 127.0.0.1 and only the gateway binds anywhere else. If they are in separate containers, put them on a private Docker network and publish only the gateway’s port. A caller who somehow reaches the engine directly bypasses every key, quota and cap you configured, so the architecture should make that impossible rather than merely discouraged.

Test it the way an attacker would: from another machine, try the engine’s port directly. Connection refused is the only acceptable answer.