{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"containers-and-driver-versions","title":"Containers, drivers and the version matrix","url":"https://addaly.com/learn/running-models-yourself/containers-and-driver-versions","position":59,"of":82,"minutes":9,"answer":"The host driver sets the ceiling and the container carries the CUDA runtime, so newer containers run on older hosts but never the reverse: verify the GPU with a bare CUDA image first.","keyIdea":"The host driver sets the ceiling and the container carries the CUDA runtime, so newer containers run on older hosts but never the reverse: verify the GPU with a bare CUDA image first, mount weights rather than baking them in, and pin the engine tag so a default cannot change underneath you."},"body":{"format":"markdown","text":"## Why the first deployment breaks and the laptop did not\n\nA model that runs on your workstation and fails on the server almost never fails because of the model. It fails because four pieces of software have to agree on a version and one of them moved: the NVIDIA kernel driver, the CUDA runtime, the PyTorch build, and the engine compiled against them.\n\nThe rule that makes sense of this is one-directional. The **driver** lives on the host and is the only part that touches the kernel. The **CUDA runtime** lives with the application, typically inside the container. A newer driver runs older runtimes; an older driver cannot run a newer runtime. So the host driver must be at least as new as the CUDA version the container expects, and upgrading the container is safe while upgrading the host driver is the thing you schedule.\n\n```bash\nnvidia-smi   # top-right shows the maximum CUDA version this driver supports\n# Driver Version: 550.54   CUDA Version: 12.4\n```\n\nThat `CUDA Version` field is the ceiling, not what is installed. A container built for CUDA 12.1 runs fine on it. A container built for CUDA 12.6 fails with a message about an insufficient driver, which people misread as a hardware problem.\n\n## Making the GPU visible inside a container\n\nDocker does not pass through a GPU by default. You need the NVIDIA Container Toolkit on the host, once:\n\n```bash\nsudo apt install -y nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\ndocker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi\n```\n\nIf that last command prints your card, everything downstream will work. If it prints `could not select device driver`, the toolkit step did not take, and no amount of fiddling with the inference image will help. Run this first on every new machine; it separates host problems from application problems in thirty seconds.\n\nAMD cards use ROCm and pass devices rather than a runtime: `--device=/dev/kfd --device=/dev/dri --group-add video`. Apple Silicon has no container GPU path at all: Docker Desktop on macOS cannot reach the Metal GPU, so a Mac runs the engine natively or not at all. That is a real constraint, not a temporary gap, and it decides the deployment shape for anyone building on a Mac Studio.\n\n## Why the image is eight gigabytes\n\nA vLLM image is large because it contains a CUDA runtime, cuDNN, PyTorch with its bundled kernels, and compiled attention kernels for several architectures. This is unavoidable and it is why you should never build weights into the image.\n\nMount them instead:\n\n```yaml\nservices:\n  vllm:\n    image: vllm/vllm-openai:v0.8.5\n    runtime: nvidia\n    ipc: host\n    volumes:\n      - ~/.cache/huggingface:/root/.cache/huggingface\n    command: >\n      --model Qwen/Qwen2.5-7B-Instruct\n      --max-model-len 8192 --gpu-memory-utilization 0.85\n      --host 0.0.0.0\n    expose: [\"8000\"]          # private network only, no ports: mapping\n```\n\nTwo details in there cost people days. `ipc: host` (or `--shm-size=8g`) is required because PyTorch's data loaders use shared memory, and Docker's default 64MB limit produces a crash that names a bus error rather than shared memory. And `expose` rather than `ports` keeps the engine reachable from the gateway container and from nothing else.\n\nPin the image tag. `latest` on an inference engine is a moving target that has, more than once, changed a default that mattered: a sampling default, a context default, a tokeniser behaviour. A pinned tag means the thing you tested is the thing that runs.\n\n## The free path without containers\n\nContainers are the right answer on a shared server and overkill on your own machine. `uv` gives you a reproducible Python environment in seconds, and llama.cpp builds from source in a couple of minutes with no Python at all. Both are free, both are honest, and a `systemd` unit around a binary is simpler to reason about than a container for a single-machine deployment. Use containers when you need to run two engines with incompatible dependencies, or when somebody else has to reproduce your setup.\n\n## Upgrade the way that lets you go back\n\nThe pattern that survives contact with users: run the new version on a second port, send it a handful of real requests, compare the outputs and the tokens-per-second against the current version, then switch the gateway's `api_base` to the new port. Keep the old container running for a day. Rolling back is then one line of configuration rather than a rebuild at the worst possible moment.\n\nWatch for the failure that upgrades produce most often: the new engine version quietly changes a default `max_model_len` or memory fraction and either refuses to start or starts with a smaller context than yesterday. Compare the startup log line by line, not just the exit code."},"check":{"question":"A server with driver 550 (reporting CUDA 12.4) runs an inference container built against CUDA 12.1 without trouble. A new container built against CUDA 12.6 refuses to start. What is the correct reading?","options":[{"text":"The container must be rebuilt against 12.4, because containers must match the host's CUDA version exactly.","correct":false,"tempting":"Over-tightens a one-directional rule into equality; the 12.1 container working on this host already disproves the exact-match claim."},{"text":"The GPU is too old for CUDA 12.6 and needs replacing.","correct":false,"tempting":"Confuses driver support with hardware capability; the same card runs 12.6 perfectly once the host driver is updated."},{"text":"CUDA compatibility runs one way, so the host driver must be updated to satisfy the newer container.","correct":true},{"text":"The NVIDIA Container Toolkit needs reconfiguring, since it mediates the runtime version between host and container.","correct":false,"tempting":"The toolkit exposes the device and driver to the container but does not translate versions, and it is already working for the 12.1 image."}]},"prev":{"slug":"a-gateway-in-front","title":"A gateway in front: keys, quotas and a bill","url":"https://addaly.com/learn/running-models-yourself/a-gateway-in-front"},"next":{"slug":"running-it-as-a-service","title":"Running it as a service that survives a reboot","url":"https://addaly.com/learn/running-models-yourself/running-it-as-a-service"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Containers, drivers and the version matrix', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/containers-and-driver-versions (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Containers, drivers and the version matrix","courseTitle":"Running Models Yourself","module":{"index":7,"slug":"serving-it-to-others","title":"Serving it to other people"},"answer":null,"keyIdea":"The host driver sets the ceiling and the container carries the CUDA runtime, so newer containers run on older hosts but never the reverse: verify the GPU with a bare CUDA image first, mount weights rather than baking them in, and pin the engine tag so a default cannot change underneath you.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-the-first-deployment-breaks-and-the-laptop-did-not","runs":[{"text":"Why the first deployment breaks and the laptop did not"}]},{"t":"p","runs":[{"text":"A model that runs on your workstation and fails on the server almost never fails because of the model. It fails because four pieces of software have to agree on a version and one of them moved: the NVIDIA kernel driver, the CUDA runtime, the PyTorch build, and the engine compiled against them."}]},{"t":"p","runs":[{"text":"The rule that makes sense of this is one-directional. The "},{"text":"driver","b":true},{"text":" lives on the host and is the only part that touches the kernel. The "},{"text":"CUDA runtime","b":true},{"text":" lives with the application, typically inside the container. A newer driver runs older runtimes; an older driver cannot run a newer runtime. So the host driver must be at least as new as the CUDA version the container expects, and upgrading the container is safe while upgrading the host driver is the thing you schedule."}]},{"t":"code","lang":"bash","label":"Bash","text":"nvidia-smi   # top-right shows the maximum CUDA version this driver supports\n# Driver Version: 550.54   CUDA Version: 12.4"},{"t":"p","runs":[{"text":"That "},{"text":"CUDA Version","code":true},{"text":" field is the ceiling, not what is installed. A container built for CUDA 12.1 runs fine on it. A container built for CUDA 12.6 fails with a message about an insufficient driver, which people misread as a hardware problem."}]},{"t":"h2","id":"making-the-gpu-visible-inside-a-container","runs":[{"text":"Making the GPU visible inside a container"}]},{"t":"p","runs":[{"text":"Docker does not pass through a GPU by default. You need the NVIDIA Container Toolkit on the host, once:"}]},{"t":"code","lang":"bash","label":"Bash","text":"sudo apt install -y nvidia-container-toolkit\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\ndocker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi"},{"t":"p","runs":[{"text":"If that last command prints your card, everything downstream will work. If it prints "},{"text":"could not select device driver","code":true},{"text":", the toolkit step did not take, and no amount of fiddling with the inference image will help. Run this first on every new machine; it separates host problems from application problems in thirty seconds."}]},{"t":"p","runs":[{"text":"AMD cards use ROCm and pass devices rather than a runtime: "},{"text":"--device=/dev/kfd --device=/dev/dri --group-add video","code":true},{"text":". Apple Silicon has no container GPU path at all: Docker Desktop on macOS cannot reach the Metal GPU, so a Mac runs the engine natively or not at all. That is a real constraint, not a temporary gap, and it decides the deployment shape for anyone building on a Mac Studio."}]},{"t":"h2","id":"why-the-image-is-eight-gigabytes","runs":[{"text":"Why the image is eight gigabytes"}]},{"t":"p","runs":[{"text":"A vLLM image is large because it contains a CUDA runtime, cuDNN, PyTorch with its bundled kernels, and compiled attention kernels for several architectures. This is unavoidable and it is why you should never build weights into the image."}]},{"t":"p","runs":[{"text":"Mount them instead:"}]},{"t":"code","lang":"yaml","label":"YAML","text":"services:\n  vllm:\n    image: vllm/vllm-openai:v0.8.5\n    runtime: nvidia\n    ipc: host\n    volumes:\n      - ~/.cache/huggingface:/root/.cache/huggingface\n    command: >\n      --model Qwen/Qwen2.5-7B-Instruct\n      --max-model-len 8192 --gpu-memory-utilization 0.85\n      --host 0.0.0.0\n    expose: [\"8000\"]          # private network only, no ports: mapping"},{"t":"p","runs":[{"text":"Two details in there cost people days. "},{"text":"ipc: host","code":true},{"text":" (or "},{"text":"--shm-size=8g","code":true},{"text":") is required because PyTorch’s data loaders use shared memory, and Docker’s default 64MB limit produces a crash that names a bus error rather than shared memory. And "},{"text":"expose","code":true},{"text":" rather than "},{"text":"ports","code":true},{"text":" keeps the engine reachable from the gateway container and from nothing else."}]},{"t":"p","runs":[{"text":"Pin the image tag. "},{"text":"latest","code":true},{"text":" on an inference engine is a moving target that has, more than once, changed a default that mattered: a sampling default, a context default, a tokeniser behaviour. A pinned tag means the thing you tested is the thing that runs."}]},{"t":"h2","id":"the-free-path-without-containers","runs":[{"text":"The free path without containers"}],"role":"free"},{"t":"p","runs":[{"text":"Containers are the right answer on a shared server and overkill on your own machine. "},{"text":"uv","code":true},{"text":" gives you a reproducible Python environment in seconds, and llama.cpp builds from source in a couple of minutes with no Python at all. Both are free, both are honest, and a "},{"text":"systemd","code":true},{"text":" unit around a binary is simpler to reason about than a container for a single-machine deployment. Use containers when you need to run two engines with incompatible dependencies, or when somebody else has to reproduce your setup."}]},{"t":"h2","id":"upgrade-the-way-that-lets-you-go-back","runs":[{"text":"Upgrade the way that lets you go back"}]},{"t":"p","runs":[{"text":"The pattern that survives contact with users: run the new version on a second port, send it a handful of real requests, compare the outputs and the tokens-per-second against the current version, then switch the gateway’s "},{"text":"api_base","code":true},{"text":" to the new port. Keep the old container running for a day. Rolling back is then one line of configuration rather than a rebuild at the worst possible moment."}]},{"t":"p","runs":[{"text":"Watch for the failure that upgrades produce most often: the new engine version quietly changes a default "},{"text":"max_model_len","code":true},{"text":" or memory fraction and either refuses to start or starts with a smaller context than yesterday. Compare the startup log line by line, not just the exit code."}]}],"check":{"question":[{"text":"A server with driver 550 (reporting CUDA 12.4) runs an inference container built against CUDA 12.1 without trouble. A new container built against CUDA 12.6 refuses to start. What is the correct reading?"}],"options":[{"text":[{"text":"The container must be rebuilt against 12.4, because containers must match the host’s CUDA version exactly."}],"correct":false,"tempting":[{"text":"Over-tightens a one-directional rule into equality; the 12.1 container working on this host already disproves the exact-match claim."}]},{"text":[{"text":"The GPU is too old for CUDA 12.6 and needs replacing."}],"correct":false,"tempting":[{"text":"Confuses driver support with hardware capability; the same card runs 12.6 perfectly once the host driver is updated."}]},{"text":[{"text":"CUDA compatibility runs one way, so the host driver must be updated to satisfy the newer container."}],"correct":true,"tempting":null},{"text":[{"text":"The NVIDIA Container Toolkit needs reconfiguring, since it mediates the runtime version between host and container."}],"correct":false,"tempting":[{"text":"The toolkit exposes the device and driver to the container but does not translate versions, and it is already working for the 12.1 image."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"a-gateway-in-front","title":"A gateway in front: keys, quotas and a bill","minutes":9},"next":{"slug":"running-it-as-a-service","title":"Running it as a service that survives a reboot","minutes":8}}}