{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"serving-it-yourself","title":"When transformers stops being the right server","url":"https://addaly.com/learn/hugging-face/serving-it-yourself","position":71,"of":75,"minutes":8,"answer":"Purpose-built servers beat a transformers loop structurally rather than by tuning (continuous batching keeps the GPU full, paged attention stops the KV cache fragmenting.","keyIdea":"Purpose-built servers beat a transformers loop structurally rather than by tuning (continuous batching keeps the GPU full, paged attention stops the KV cache fragmenting, and prefix caching reuses a shared system prompt) and unlike at training time, quantization under these stacks buys speed as well as memory."},"body":{"format":"markdown","text":"## The wall you hit\n\nA Flask endpoint wrapping `model.generate` works, and it works for one user at a time. Requests queue, the GPU sits idle between them, and at ten concurrent users latency becomes unacceptable while utilisation stays under a third.\n\nThe cause is structural rather than a tuning problem. A naive server processes one batch to completion before starting the next, so a request that arrives one token into a 500-token generation waits for all 500. And the KV cache is allocated as a contiguous block sized for the worst case, so most of the memory reserved for it is never used.\n\n## What purpose-built servers do differently\n\n**Continuous batching.** New requests join the running batch at the next token step, and finished ones leave immediately. The GPU stays full. This alone is typically a several-fold throughput gain at the same latency.\n\n**Paged attention.** The KV cache is allocated in fixed-size blocks, like operating-system memory pages, rather than one contiguous reservation per sequence. Fragmentation drops sharply and far more concurrent sequences fit in the same memory.\n\n**Prefix caching.** When many requests share an opening (a long system prompt, a retrieved document) the computation for that shared prefix is done once and reused. For a retrieval-augmented application where every request carries the same instructions, this is a large and easy saving.\n\n```bash\npip install vllm\nvllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192\n```\n\nThat gives an OpenAI-compatible server on port 8000. Text-generation-inference is Hugging Face's equivalent, distributed as a container:\n\n```bash\ndocker run --gpus all -p 8080:80 \\\n  ghcr.io/huggingface/text-generation-inference:latest \\\n  --model-id Qwen/Qwen2.5-7B-Instruct\n```\n\nBoth are free and open source. Both expose the same API shape as the providers in the last lesson, so your client code does not change.\n\n## The CPU path is the same idea\n\n```bash\nllama-server -hf unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080 -c 8192\n```\n\nllama.cpp's server also does continuous batching and prefix caching, on a CPU or Apple silicon, with no GPU at all. For a small team serving modest internal traffic this is frequently the whole answer: one machine, one binary, no cloud account.\n\n## What you give up\n\n- **Flexibility.** These servers run generation. A custom head, an intermediate layer, a bespoke loss: none of that fits, and transformers remains the right tool for it.\n- **Model coverage.** Support is per-architecture and a brand-new model may not be supported for weeks.\n- **Startup time.** Loading and warming a large model takes minutes, which matters if you were hoping to scale to zero.\n- **Operational surface.** Now you own a service: monitoring, restarts, upgrades, capacity.\n\n## Quantization and serving, together\n\nServing stacks load AWQ, GPTQ and native low-precision formats with kernels written for them, so unlike the bitsandbytes path from module three, quantization here buys **speed as well as memory**. A 4-bit AWQ model under vLLM is faster than bfloat16, not merely smaller. That is the reverse of the training-time picture and worth holding separately in your head.\n\n## What to measure before and after\n\nThe same four numbers as module three, plus two that only appear under load:\n\n- **Throughput** in tokens per second, aggregated across concurrent requests.\n- **Time to first token** at the 95th percentile, not the mean.\n- **Concurrency at which latency degrades**: the number that tells you when to add a second machine.\n- **GPU utilisation.** If it sits at 30% under load, the server is the bottleneck rather than the hardware, and that is the finding that justifies this whole lesson.\n\nLoad-test with something that issues concurrent requests rather than a loop. A sequential loop measures latency and tells you nothing about the problem these servers exist to solve.\n\n## Do this now\n\nTake a model you serve and run the same twenty concurrent requests against a plain transformers loop and against vLLM or llama-server. Record aggregate throughput for both. The ratio is usually large enough that the decision makes itself."},"check":{"question":"A GPU shows 30% utilisation while users complain about latency under concurrent load. What does this most likely indicate?","options":[{"text":"The model is too large for the card, so layers are being swapped between GPU and CPU memory each step.","correct":false,"tempting":"Offloading would show high utilisation punctuated by transfer stalls, and would be slow even for a single request."},{"text":"Requests run batch-by-batch to completion, so the GPU idles while new ones wait.","correct":true},{"text":"The KV cache has exhausted available memory, forcing the server to recompute attention for earlier tokens.","correct":false,"tempting":"Cache exhaustion produces errors or evictions rather than low utilisation."},{"text":"Tokenization on the CPU is the bottleneck, since it cannot keep the GPU supplied at this request rate.","correct":false,"tempting":"Tokenization is negligible against generation and would not hold a GPU at 30%."}]},"prev":{"slug":"inference-providers","title":"Renting the compute instead of owning it","url":"https://addaly.com/learn/hugging-face/inference-providers"},"next":{"slug":"publishing-your-own","title":"Publish it so a stranger can use it without writing to you","url":"https://addaly.com/learn/hugging-face/publishing-your-own"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When transformers stops being the right server', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/serving-it-yourself (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When transformers stops being the right server","courseTitle":"Hugging Face, End to End","module":{"index":8,"slug":"depending-on-it-and-publishing","title":"Depending on it, and publishing so others can"},"answer":null,"keyIdea":"Purpose-built servers beat a transformers loop structurally rather than by tuning (continuous batching keeps the GPU full, paged attention stops the KV cache fragmenting, and prefix caching reuses a shared system prompt) and unlike at training time, quantization under these stacks buys speed as well as memory.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-wall-you-hit","runs":[{"text":"The wall you hit"}]},{"t":"p","runs":[{"text":"A Flask endpoint wrapping "},{"text":"model.generate","code":true},{"text":" works, and it works for one user at a time. Requests queue, the GPU sits idle between them, and at ten concurrent users latency becomes unacceptable while utilisation stays under a third."}]},{"t":"p","runs":[{"text":"The cause is structural rather than a tuning problem. A naive server processes one batch to completion before starting the next, so a request that arrives one token into a 500-token generation waits for all 500. And the KV cache is allocated as a contiguous block sized for the worst case, so most of the memory reserved for it is never used."}]},{"t":"h2","id":"what-purpose-built-servers-do-differently","runs":[{"text":"What purpose-built servers do differently"}]},{"t":"p","runs":[{"text":"Continuous batching.","b":true},{"text":" New requests join the running batch at the next token step, and finished ones leave immediately. The GPU stays full. This alone is typically a several-fold throughput gain at the same latency."}]},{"t":"p","runs":[{"text":"Paged attention.","b":true},{"text":" The KV cache is allocated in fixed-size blocks, like operating-system memory pages, rather than one contiguous reservation per sequence. Fragmentation drops sharply and far more concurrent sequences fit in the same memory."}]},{"t":"p","runs":[{"text":"Prefix caching.","b":true},{"text":" When many requests share an opening (a long system prompt, a retrieved document) the computation for that shared prefix is done once and reused. For a retrieval-augmented application where every request carries the same instructions, this is a large and easy saving."}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install vllm\nvllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192"},{"t":"p","runs":[{"text":"That gives an OpenAI-compatible server on port 8000. Text-generation-inference is Hugging Face’s equivalent, distributed as a container:"}]},{"t":"code","lang":"bash","label":"Bash","text":"docker run --gpus all -p 8080:80 \\\n  ghcr.io/huggingface/text-generation-inference:latest \\\n  --model-id Qwen/Qwen2.5-7B-Instruct"},{"t":"p","runs":[{"text":"Both are free and open source. Both expose the same API shape as the providers in the last lesson, so your client code does not change."}]},{"t":"h2","id":"the-cpu-path-is-the-same-idea","runs":[{"text":"The CPU path is the same idea"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -hf unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080 -c 8192"},{"t":"p","runs":[{"text":"llama.cpp’s server also does continuous batching and prefix caching, on a CPU or Apple silicon, with no GPU at all. For a small team serving modest internal traffic this is frequently the whole answer: one machine, one binary, no cloud account."}]},{"t":"h2","id":"what-you-give-up","runs":[{"text":"What you give up"}]},{"t":"ul","items":[[{"text":"Flexibility.","b":true},{"text":" These servers run generation. A custom head, an intermediate layer, a bespoke loss: none of that fits, and transformers remains the right tool for it."}],[{"text":"Model coverage.","b":true},{"text":" Support is per-architecture and a brand-new model may not be supported for weeks."}],[{"text":"Startup time.","b":true},{"text":" Loading and warming a large model takes minutes, which matters if you were hoping to scale to zero."}],[{"text":"Operational surface.","b":true},{"text":" Now you own a service: monitoring, restarts, upgrades, capacity."}]]},{"t":"h2","id":"quantization-and-serving-together","runs":[{"text":"Quantization and serving, together"}]},{"t":"p","runs":[{"text":"Serving stacks load AWQ, GPTQ and native low-precision formats with kernels written for them, so unlike the bitsandbytes path from module three, quantization here buys "},{"text":"speed as well as memory","b":true},{"text":". A 4-bit AWQ model under vLLM is faster than bfloat16, not merely smaller. That is the reverse of the training-time picture and worth holding separately in your head."}]},{"t":"h2","id":"what-to-measure-before-and-after","runs":[{"text":"What to measure before and after"}]},{"t":"p","runs":[{"text":"The same four numbers as module three, plus two that only appear under load:"}]},{"t":"ul","items":[[{"text":"Throughput","b":true},{"text":" in tokens per second, aggregated across concurrent requests."}],[{"text":"Time to first token","b":true},{"text":" at the 95th percentile, not the mean."}],[{"text":"Concurrency at which latency degrades","b":true},{"text":": the number that tells you when to add a second machine."}],[{"text":"GPU utilisation.","b":true},{"text":" If it sits at 30% under load, the server is the bottleneck rather than the hardware, and that is the finding that justifies this whole lesson."}]]},{"t":"p","runs":[{"text":"Load-test with something that issues concurrent requests rather than a loop. A sequential loop measures latency and tells you nothing about the problem these servers exist to solve."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Take a model you serve and run the same twenty concurrent requests against a plain transformers loop and against vLLM or llama-server. Record aggregate throughput for both. The ratio is usually large enough that the decision makes itself."}]}],"check":{"question":[{"text":"A GPU shows 30% utilisation while users complain about latency under concurrent load. What does this most likely indicate?"}],"options":[{"text":[{"text":"The model is too large for the card, so layers are being swapped between GPU and CPU memory each step."}],"correct":false,"tempting":[{"text":"Offloading would show high utilisation punctuated by transfer stalls, and would be slow even for a single request."}]},{"text":[{"text":"Requests run batch-by-batch to completion, so the GPU idles while new ones wait."}],"correct":true,"tempting":null},{"text":[{"text":"The KV cache has exhausted available memory, forcing the server to recompute attention for earlier tokens."}],"correct":false,"tempting":[{"text":"Cache exhaustion produces errors or evictions rather than low utilisation."}]},{"text":[{"text":"Tokenization on the CPU is the bottleneck, since it cannot keep the GPU supplied at this request rate."}],"correct":false,"tempting":[{"text":"Tokenization is negligible against generation and would not hold a GPU at 30%."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"inference-providers","title":"Renting the compute instead of owning it","minutes":8},"next":{"slug":"publishing-your-own","title":"Publish it so a stranger can use it without writing to you","minutes":10}}}