Containers, drivers and the version matrix

Why the first deployment breaks and the laptop did not

A model that runs on your workstation and fails on the server almost never fails because of the model. It fails because four pieces of software have to agree on a version and one of them moved: the NVIDIA kernel driver, the CUDA runtime, the PyTorch build, and the engine compiled against them.

The rule that makes sense of this is one-directional. The driver lives on the host and is the only part that touches the kernel. The CUDA runtime lives with the application, typically inside the container. A newer driver runs older runtimes; an older driver cannot run a newer runtime. So the host driver must be at least as new as the CUDA version the container expects, and upgrading the container is safe while upgrading the host driver is the thing you schedule.

nvidia-smi   # top-right shows the maximum CUDA version this driver supports
# Driver Version: 550.54   CUDA Version: 12.4

That CUDA Version field is the ceiling, not what is installed. A container built for CUDA 12.1 runs fine on it. A container built for CUDA 12.6 fails with a message about an insufficient driver, which people misread as a hardware problem.

Making the GPU visible inside a container

Docker does not pass through a GPU by default. You need the NVIDIA Container Toolkit on the host, once:

sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If that last command prints your card, everything downstream will work. If it prints could not select device driver, the toolkit step did not take, and no amount of fiddling with the inference image will help. Run this first on every new machine; it separates host problems from application problems in thirty seconds.

AMD cards use ROCm and pass devices rather than a runtime: --device=/dev/kfd --device=/dev/dri --group-add video. Apple Silicon has no container GPU path at all: Docker Desktop on macOS cannot reach the Metal GPU, so a Mac runs the engine natively or not at all. That is a real constraint, not a temporary gap, and it decides the deployment shape for anyone building on a Mac Studio.

Why the image is eight gigabytes

A vLLM image is large because it contains a CUDA runtime, cuDNN, PyTorch with its bundled kernels, and compiled attention kernels for several architectures. This is unavoidable and it is why you should never build weights into the image.

Mount them instead:

services:
  vllm:
    image: vllm/vllm-openai:v0.8.5
    runtime: nvidia
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model Qwen/Qwen2.5-7B-Instruct
      --max-model-len 8192 --gpu-memory-utilization 0.85
      --host 0.0.0.0
    expose: ["8000"]          # private network only, no ports: mapping

Two details in there cost people days. ipc: host (or --shm-size=8g) is required because PyTorch’s data loaders use shared memory, and Docker’s default 64MB limit produces a crash that names a bus error rather than shared memory. And expose rather than ports keeps the engine reachable from the gateway container and from nothing else.

Pin the image tag. latest on an inference engine is a moving target that has, more than once, changed a default that mattered: a sampling default, a context default, a tokeniser behaviour. A pinned tag means the thing you tested is the thing that runs.

The free path without containers

Containers are the right answer on a shared server and overkill on your own machine. uv gives you a reproducible Python environment in seconds, and llama.cpp builds from source in a couple of minutes with no Python at all. Both are free, both are honest, and a systemd unit around a binary is simpler to reason about than a container for a single-machine deployment. Use containers when you need to run two engines with incompatible dependencies, or when somebody else has to reproduce your setup.

Upgrade the way that lets you go back

The pattern that survives contact with users: run the new version on a second port, send it a handful of real requests, compare the outputs and the tokens-per-second against the current version, then switch the gateway’s api_base to the new port. Keep the old container running for a day. Rolling back is then one line of configuration rather than a rebuild at the worst possible moment.

Watch for the failure that upgrades produce most often: the new engine version quietly changes a default max_model_len or memory fraction and either refuses to start or starts with a smaller context than yesterday. Compare the startup log line by line, not just the exit code.