When transformers stops being the right server

The wall you hit

A Flask endpoint wrapping model.generate works, and it works for one user at a time. Requests queue, the GPU sits idle between them, and at ten concurrent users latency becomes unacceptable while utilisation stays under a third.

The cause is structural rather than a tuning problem. A naive server processes one batch to completion before starting the next, so a request that arrives one token into a 500-token generation waits for all 500. And the KV cache is allocated as a contiguous block sized for the worst case, so most of the memory reserved for it is never used.

What purpose-built servers do differently

Continuous batching. New requests join the running batch at the next token step, and finished ones leave immediately. The GPU stays full. This alone is typically a several-fold throughput gain at the same latency.

Paged attention. The KV cache is allocated in fixed-size blocks, like operating-system memory pages, rather than one contiguous reservation per sequence. Fragmentation drops sharply and far more concurrent sequences fit in the same memory.

Prefix caching. When many requests share an opening (a long system prompt, a retrieved document) the computation for that shared prefix is done once and reused. For a retrieval-augmented application where every request carries the same instructions, this is a large and easy saving.

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192

That gives an OpenAI-compatible server on port 8000. Text-generation-inference is Hugging Face’s equivalent, distributed as a container:

docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id Qwen/Qwen2.5-7B-Instruct

Both are free and open source. Both expose the same API shape as the providers in the last lesson, so your client code does not change.

The CPU path is the same idea

llama-server -hf unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080 -c 8192

llama.cpp’s server also does continuous batching and prefix caching, on a CPU or Apple silicon, with no GPU at all. For a small team serving modest internal traffic this is frequently the whole answer: one machine, one binary, no cloud account.

What you give up

• Flexibility. These servers run generation. A custom head, an intermediate layer, a bespoke loss: none of that fits, and transformers remains the right tool for it.
• Model coverage. Support is per-architecture and a brand-new model may not be supported for weeks.
• Startup time. Loading and warming a large model takes minutes, which matters if you were hoping to scale to zero.
• Operational surface. Now you own a service: monitoring, restarts, upgrades, capacity.

Quantization and serving, together

Serving stacks load AWQ, GPTQ and native low-precision formats with kernels written for them, so unlike the bitsandbytes path from module three, quantization here buys speed as well as memory. A 4-bit AWQ model under vLLM is faster than bfloat16, not merely smaller. That is the reverse of the training-time picture and worth holding separately in your head.

What to measure before and after

The same four numbers as module three, plus two that only appear under load:

• Throughput in tokens per second, aggregated across concurrent requests.
• Time to first token at the 95th percentile, not the mean.
• Concurrency at which latency degrades: the number that tells you when to add a second machine.
• GPU utilisation. If it sits at 30% under load, the server is the bottleneck rather than the hardware, and that is the finding that justifies this whole lesson.

Load-test with something that issues concurrent requests rather than a loop. A sequential loop measures latency and tells you nothing about the problem these servers exist to solve.

Do this now

Take a model you serve and run the same twenty concurrent requests against a plain transformers loop and against vLLM or llama-server. Record aggregate throughput for both. The ratio is usually large enough that the decision makes itself.