{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"a-gateway-in-front","title":"A gateway in front: keys, quotas and a bill","url":"https://addaly.com/learn/running-models-yourself/a-gateway-in-front","position":58,"of":82,"minutes":9,"answer":"Inference engines implement inference and nothing else, so identity, quotas and audit belong in a gateway that holds per-person keys, enforces requests, tokens and concurrency as three separate limits.","keyIdea":"Inference engines implement inference and nothing else, so identity, quotas and audit belong in a gateway that holds per-person keys, enforces requests, tokens and concurrency as three separate limits, caps output length where the caller cannot raise it, and forwards only the inference paths."},"body":{"format":"markdown","text":"## What the engine deliberately does not do\n\nvLLM, llama.cpp's server and Ollama all implement inference and almost nothing else. vLLM has a single `--api-key` shared secret, which is better than nothing and worse than useless the moment more than one person holds it: you cannot tell who spent the GPU hour, and revoking it locks out everybody. Ollama has no authentication at all. This is not an oversight. Identity, quotas, billing and audit belong in a layer that outlives whichever engine you are running this month.\n\nThat layer is a gateway: a small service that sits between callers and the engine, holds the keys, counts the tokens and forwards the request. Because every engine speaks the same OpenAI-shaped API, the gateway is engine-agnostic.\n\n## The free option that does the most\n\nLiteLLM Proxy is open source under MIT and runs as one container. It gives you virtual keys per person or per application, budgets in currency and in tokens, per-key rate limits, request logs, and a model-name mapping so callers ask for `internal-fast` while you decide whether that is a local Qwen or a hosted model this week.\n\n```yaml\n# config.yaml\nmodel_list:\n  - model_name: internal-fast\n    litellm_params:\n      model: openai/qwen2.5-7b-instruct\n      api_base: http://127.0.0.1:8000/v1\n      api_key: dummy\nlitellm_settings:\n  max_budget: 40          # currency units per month, all keys\n  budget_duration: 30d\n```\n\n```bash\ndocker run -p 4000:4000 -v $(pwd)/config.yaml:/app/config.yaml \\\n  -e LITELLM_MASTER_KEY=sk-... ghcr.io/berriai/litellm:main-latest \\\n  --config /app/config.yaml\n\n# then mint a key with its own ceiling\ncurl -X POST http://127.0.0.1:4000/key/generate \\\n  -H 'Authorization: Bearer sk-...' -H 'Content-Type: application/json' \\\n  -d '{\"models\":[\"internal-fast\"],\"max_budget\":5,\"rpm_limit\":20,\"metadata\":{\"user\":\"priya\"}}'\n```\n\nThe key that comes back is the only thing a caller ever sees. Delete it and that person is out, and nobody else notices.\n\nThe smaller alternative is a reverse proxy you already understand. Caddy in about six lines will terminate TLS with an automatically issued certificate and check a bearer token; nginx does the same with `auth_request` or a map of tokens. You get authentication and TLS but no accounting, which is fine when the whole audience is three colleagues.\n\n## Rate limits are three different limits\n\nPeople write one number and are surprised when the machine still falls over.\n\n> Figure: What a gateway holds that the engine deliberately does not. Layers, top to bottom: Requests per minute (Stops a loop in a script. Does not protect the card); Tokens per minute (The limit that maps to the hardware); Concurrency, per key (Stops one batch job filling every slot); A maximum output length (A ceiling the caller cannot raise); An allow-list of paths (Inference forwarded; model pull and delete refused). Engines implement inference and almost nothing else, and that is not an oversight: identity, quotas and audit outlive whichever engine you are running this month. With a gateway in place the engine binds to loopback, and a caller who reaches it directly would bypass every limit here.\n\n- **Requests per minute** protects against a loop in somebody's script. Cheap to enforce, and it does not protect the GPU, because one request can be enormous.\n- **Tokens per minute** protects the GPU. This is the limit that maps to the hardware, because throughput is tokens, not calls.\n- **Concurrency** (how many of one caller's requests may be in flight at once) protects everybody else's latency. Without it, one batch job with 200 parallel workers fills the engine's queue and every interactive user waits behind it.\n\nSet all three. A reasonable starting point for an internal service on one card: 60 requests per minute, tokens per minute at roughly half your measured sustained throughput, and a concurrency of four per key.\n\n## Set the maximum output length at the gateway\n\nThe single most effective protection is a cap the caller cannot raise. A request with no `max_tokens` and a model that will not stop occupies a slot until it hits the context limit: on a 32k context that is minutes of exclusive GPU time for one caller. Enforce a default and a ceiling in the gateway, and let callers request less but never more.\n\n> Most \"the server hung\" reports are one runaway generation holding a slot while everything else queues behind it.\n\n## What the gateway should refuse to pass through\n\nEngines expose management endpoints that have no business being reachable by callers. Ollama's API can pull a model from the internet onto your disk (`/api/pull`), delete one (`/api/delete`) and create new ones (`/api/create`). vLLM can expose a metrics endpoint and, if enabled, adapter loading. Allow-list the paths the gateway forwards (chat completions, completions, embeddings, models) and drop everything else. This is one line of configuration and it removes a whole category of problem.\n\n## Keep the engine on loopback\n\nWith a gateway in place the engine binds to `127.0.0.1` and only the gateway binds anywhere else. If they are in separate containers, put them on a private Docker network and publish only the gateway's port. A caller who somehow reaches the engine directly bypasses every key, quota and cap you configured, so the architecture should make that impossible rather than merely discouraged.\n\nTest it the way an attacker would: from another machine, try the engine's port directly. Connection refused is the only acceptable answer."},"check":{"question":"An internal endpoint has a 60-requests-per-minute limit per key, yet one team's batch job still makes the chat interface unusably slow. What is the mechanism?","options":[{"text":"The request limit is being applied per IP address rather than per key, so the batch job's requests are not being counted.","correct":false,"tempting":"Assumes a misconfiguration when the described limit is working exactly as specified; the problem is what the limit measures, not where it is applied."},{"text":"Rate limits must be enforced by the engine itself; a gateway can only observe traffic and cannot delay it.","correct":false,"tempting":"Gateways do enforce limits by queueing or rejecting, which is precisely why they are the right place for the token and concurrency limits this setup is missing."},{"text":"The engine is not using continuous batching, so requests are processed strictly one after another.","correct":false,"tempting":"Would make the queue worse but is not what the limit failed to prevent; even with continuous batching there is a fixed number of slots, and long generations occupy them."},{"text":"A request limit counts calls, not work, so a few long generations hold every engine slot while short ones queue.","correct":true}]},"prev":{"slug":"reaching-the-machine-safely","title":"Reaching the machine from somewhere else","url":"https://addaly.com/learn/running-models-yourself/reaching-the-machine-safely"},"next":{"slug":"containers-and-driver-versions","title":"Containers, drivers and the version matrix","url":"https://addaly.com/learn/running-models-yourself/containers-and-driver-versions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A gateway in front: keys, quotas and a bill', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/a-gateway-in-front (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A gateway in front: keys, quotas and a bill","courseTitle":"Running Models Yourself","module":{"index":7,"slug":"serving-it-to-others","title":"Serving it to other people"},"answer":null,"keyIdea":"Inference engines implement inference and nothing else, so identity, quotas and audit belong in a gateway that holds per-person keys, enforces requests, tokens and concurrency as three separate limits, caps output length where the caller cannot raise it, and forwards only the inference paths.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"What a gateway holds that the engine deliberately does not","layers":[{"label":"Requests per minute","detail":"Stops a loop in a script. Does not protect the card"},{"label":"Tokens per minute","detail":"The limit that maps to the hardware"},{"label":"Concurrency, per key","detail":"Stops one batch job filling every slot"},{"label":"A maximum output length","detail":"A ceiling the caller cannot raise"},{"label":"An allow-list of paths","detail":"Inference forwarded; model pull and delete refused"}],"caption":"Engines implement inference and almost nothing else, and that is not an oversight: identity, quotas and audit outlive whichever engine you are running this month. With a gateway in place the engine binds to loopback, and a caller who reaches it directly would bypass every limit here."},"alt":"Figure: What a gateway holds that the engine deliberately does not. Layers, top to bottom: Requests per minute (Stops a loop in a script. Does not protect the card); Tokens per minute (The limit that maps to the hardware); Concurrency, per key (Stops one batch job filling every slot); A maximum output length (A ceiling the caller cannot raise); An allow-list of paths (Inference forwarded; model pull and delete refused). Engines implement inference and almost nothing else, and that is not an oversight: identity, quotas and audit outlive whichever engine you are running this month. With a gateway in place the engine binds to loopback, and a caller who reaches it directly would bypass every limit here."}},"blocks":[{"t":"h2","id":"what-the-engine-deliberately-does-not-do","runs":[{"text":"What the engine deliberately does not do"}]},{"t":"p","runs":[{"text":"vLLM, llama.cpp’s server and Ollama all implement inference and almost nothing else. vLLM has a single "},{"text":"--api-key","code":true},{"text":" shared secret, which is better than nothing and worse than useless the moment more than one person holds it: you cannot tell who spent the GPU hour, and revoking it locks out everybody. Ollama has no authentication at all. This is not an oversight. Identity, quotas, billing and audit belong in a layer that outlives whichever engine you are running this month."}]},{"t":"p","runs":[{"text":"That layer is a gateway: a small service that sits between callers and the engine, holds the keys, counts the tokens and forwards the request. Because every engine speaks the same OpenAI-shaped API, the gateway is engine-agnostic."}]},{"t":"h2","id":"the-free-option-that-does-the-most","runs":[{"text":"The free option that does the most"}]},{"t":"p","runs":[{"text":"LiteLLM Proxy is open source under MIT and runs as one container. It gives you virtual keys per person or per application, budgets in currency and in tokens, per-key rate limits, request logs, and a model-name mapping so callers ask for "},{"text":"internal-fast","code":true},{"text":" while you decide whether that is a local Qwen or a hosted model this week."}]},{"t":"code","lang":"yaml","label":"YAML","text":"# config.yaml\nmodel_list:\n  - model_name: internal-fast\n    litellm_params:\n      model: openai/qwen2.5-7b-instruct\n      api_base: http://127.0.0.1:8000/v1\n      api_key: dummy\nlitellm_settings:\n  max_budget: 40          # currency units per month, all keys\n  budget_duration: 30d"},{"t":"code","lang":"bash","label":"Bash","text":"docker run -p 4000:4000 -v $(pwd)/config.yaml:/app/config.yaml \\\n  -e LITELLM_MASTER_KEY=sk-... ghcr.io/berriai/litellm:main-latest \\\n  --config /app/config.yaml\n\n# then mint a key with its own ceiling\ncurl -X POST http://127.0.0.1:4000/key/generate \\\n  -H 'Authorization: Bearer sk-...' -H 'Content-Type: application/json' \\\n  -d '{\"models\":[\"internal-fast\"],\"max_budget\":5,\"rpm_limit\":20,\"metadata\":{\"user\":\"priya\"}}'"},{"t":"p","runs":[{"text":"The key that comes back is the only thing a caller ever sees. Delete it and that person is out, and nobody else notices."}]},{"t":"p","runs":[{"text":"The smaller alternative is a reverse proxy you already understand. Caddy in about six lines will terminate TLS with an automatically issued certificate and check a bearer token; nginx does the same with "},{"text":"auth_request","code":true},{"text":" or a map of tokens. You get authentication and TLS but no accounting, which is fine when the whole audience is three colleagues."}]},{"t":"h2","id":"rate-limits-are-three-different-limits","runs":[{"text":"Rate limits are three different limits"}]},{"t":"p","runs":[{"text":"People write one number and are surprised when the machine still falls over."}]},{"t":"ul","items":[[{"text":"Requests per minute","b":true},{"text":" protects against a loop in somebody’s script. Cheap to enforce, and it does not protect the GPU, because one request can be enormous."}],[{"text":"Tokens per minute","b":true},{"text":" protects the GPU. This is the limit that maps to the hardware, because throughput is tokens, not calls."}],[{"text":"Concurrency","b":true},{"text":" (how many of one caller’s requests may be in flight at once) protects everybody else’s latency. Without it, one batch job with 200 parallel workers fills the engine’s queue and every interactive user waits behind it."}]]},{"t":"p","runs":[{"text":"Set all three. A reasonable starting point for an internal service on one card: 60 requests per minute, tokens per minute at roughly half your measured sustained throughput, and a concurrency of four per key."}]},{"t":"h2","id":"set-the-maximum-output-length-at-the-gateway","runs":[{"text":"Set the maximum output length at the gateway"}]},{"t":"p","runs":[{"text":"The single most effective protection is a cap the caller cannot raise. A request with no "},{"text":"max_tokens","code":true},{"text":" and a model that will not stop occupies a slot until it hits the context limit: on a 32k context that is minutes of exclusive GPU time for one caller. Enforce a default and a ceiling in the gateway, and let callers request less but never more."}]},{"t":"quote","runs":[{"text":"Most “the server hung” reports are one runaway generation holding a slot while everything else queues behind it."}]},{"t":"h2","id":"what-the-gateway-should-refuse-to-pass-through","runs":[{"text":"What the gateway should refuse to pass through"}]},{"t":"p","runs":[{"text":"Engines expose management endpoints that have no business being reachable by callers. Ollama’s API can pull a model from the internet onto your disk ("},{"text":"/api/pull","code":true},{"text":"), delete one ("},{"text":"/api/delete","code":true},{"text":") and create new ones ("},{"text":"/api/create","code":true},{"text":"). vLLM can expose a metrics endpoint and, if enabled, adapter loading. Allow-list the paths the gateway forwards (chat completions, completions, embeddings, models) and drop everything else. This is one line of configuration and it removes a whole category of problem."}]},{"t":"h2","id":"keep-the-engine-on-loopback","runs":[{"text":"Keep the engine on loopback"}]},{"t":"p","runs":[{"text":"With a gateway in place the engine binds to "},{"text":"127.0.0.1","code":true},{"text":" and only the gateway binds anywhere else. If they are in separate containers, put them on a private Docker network and publish only the gateway’s port. A caller who somehow reaches the engine directly bypasses every key, quota and cap you configured, so the architecture should make that impossible rather than merely discouraged."}]},{"t":"p","runs":[{"text":"Test it the way an attacker would: from another machine, try the engine’s port directly. Connection refused is the only acceptable answer."}]}],"check":{"question":[{"text":"An internal endpoint has a 60-requests-per-minute limit per key, yet one team’s batch job still makes the chat interface unusably slow. What is the mechanism?"}],"options":[{"text":[{"text":"The request limit is being applied per IP address rather than per key, so the batch job’s requests are not being counted."}],"correct":false,"tempting":[{"text":"Assumes a misconfiguration when the described limit is working exactly as specified; the problem is what the limit measures, not where it is applied."}]},{"text":[{"text":"Rate limits must be enforced by the engine itself; a gateway can only observe traffic and cannot delay it."}],"correct":false,"tempting":[{"text":"Gateways do enforce limits by queueing or rejecting, which is precisely why they are the right place for the token and concurrency limits this setup is missing."}]},{"text":[{"text":"The engine is not using continuous batching, so requests are processed strictly one after another."}],"correct":false,"tempting":[{"text":"Would make the queue worse but is not what the limit failed to prevent; even with continuous batching there is a fixed number of slots, and long generations occupy them."}]},{"text":[{"text":"A request limit counts calls, not work, so a few long generations hold every engine slot while short ones queue."}],"correct":true,"tempting":null}],"order":[1,3,2,0],"why":null},"prev":{"slug":"reaching-the-machine-safely","title":"Reaching the machine from somewhere else","minutes":9},"next":{"slug":"containers-and-driver-versions","title":"Containers, drivers and the version matrix","minutes":9}}}