{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"installing-the-right-build","title":"Installing the right build for your hardware","url":"https://addaly.com/learn/running-models-yourself/installing-the-right-build","position":30,"of":82,"minutes":9,"answer":"The build has to match the accelerator, and a binary compiled without your backend runs silently on the CPU. Verify with a benchmark that names the device before configuring anything else.","keyIdea":"The build has to match the accelerator, and a binary compiled without your backend runs silently on the CPU. Verify with a benchmark that names the device before configuring anything else."},"body":{"format":"markdown","text":"## The commonest local-inference bug\n\nA model runs, produces sensible text, and is thirty times slower than it should be. The cause, nearly always: the binary was compiled without support for your accelerator, so everything is on the CPU. `nvidia-smi` shows 0% utilisation while the model is generating. Check that first, before any configuration.\n\nThe backend has to match your hardware, and there are five that matter.\n\n**CUDA**: NVIDIA. The best-supported path by a wide margin. Needs a driver new enough for the CUDA version the build was compiled against; the runtime ships with the build, so you do not need the full toolkit unless you are compiling.\n\n**ROCm**: AMD. Works well on supported cards, and the list of supported cards is shorter than the list of cards AMD sells. Check yours before planning around it. `HSA_OVERRIDE_GFX_VERSION` is the environment variable that makes several unofficially supported cards work, and it is a well-trodden path with rough edges.\n\n**Metal**: Apple Silicon. Built in, no configuration, works.\n\n**Vulkan**: anything with a graphics driver, including Intel and AMD integrated graphics. Slower than the native backends and enormously faster than CPU-only for prompt processing. This is the backend most laptop owners should be using and most are not.\n\n**CPU**: always available, and the fallback everything silently lands on.\n\n## Getting a working build without compiling\n\nThe pre-built releases cover most people:\n\n```bash\n# Ollama, all platforms, picks its own backend\ncurl -fsSL https://ollama.com/install.sh | sh\n\n# llama.cpp on macOS\nbrew install llama.cpp\n\n# llama.cpp elsewhere: download a release archive matching your backend\n# from the project's releases page, e.g. -cuda-, -vulkan-, -hip-\n```\n\nVerify immediately, before doing anything else:\n\n```bash\nllama-cli --version           # should name the backend\nllama-bench -m any-model.gguf -p 128 -n 32\n```\n\nIf `llama-bench` reports a device that is not your GPU, stop and fix the build. Everything downstream is wasted effort until this is right.\n\n## Compiling, when it is worth it\n\nWorth it for: a CPU with AVX-512 or AMX, an unusual AMD card, or a bleeding-edge model whose architecture is only in the main branch.\n\n```bash\ngit clone https://github.com/ggml-org/llama.cpp\ncmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release\ncmake --build build --config Release -j\n```\n\nSwap `-DGGML_CUDA=ON` for `-DGGML_VULKAN=ON`, `-DGGML_HIP=ON` or `-DGGML_METAL=ON`. Twenty minutes on a modern machine, and a native build can beat a generic one by a useful margin on prefill.\n\n## Python environments, and the reason to isolate them\n\nvLLM, transformers and the training tools bring large, version-sensitive dependency trees. Installing them into your system Python will eventually break something else you rely on. Use an isolated environment, and prefer a fast resolver:\n\n```bash\npip install uv\nuv venv --python 3.12 && source .venv/bin/activate\nuv pip install vllm\n```\n\nvLLM in particular pins a specific PyTorch build against a specific CUDA version. Fighting that is a bad afternoon; a fresh environment per project is the cheap way out.\n\n## Docker, and what it does not solve\n\nContainers give you a known-good combination of driver-adjacent libraries, which removes most version arguments:\n\n```bash\ndocker run --gpus all -p 8000:8000 \\\n  -v ~/models:/models \\\n  vllm/vllm-openai:latest --model /models/my-model\n```\n\nThe host driver still has to be new enough: the container carries the runtime, not the kernel driver. And on Windows, GPU passthrough works through WSL2 with a Windows-side NVIDIA driver, not a driver installed inside the Linux distribution. Installing one there is a common and confusing mistake.\n\n## A working order that avoids most trouble\n\n1. Update the graphics driver.\n2. Install Ollama and run a small model. Confirm the GPU is used.\n3. Only then install anything more specialised, in its own environment.\n4. Record the versions that worked (driver, engine, CUDA or ROCm) in a text file.\n\nThat last step sounds fussy until an upgrade breaks something and you need to know what changed. Engines in this field move weekly, and regressions are normal rather than exceptional."},"check":{"question":"On a Windows laptop with an NVIDIA GPU, someone installs a Linux NVIDIA driver inside their WSL2 distribution, then runs a CUDA build of llama.cpp there. It runs but shows no GPU use. What is wrong?","options":[{"text":"WSL2 cannot pass through a discrete GPU at all; CUDA workloads must run on Windows directly or in a virtual machine with device passthrough.","correct":false,"tempting":"Denies a capability that exists and works well, leading to an unnecessary and expensive change of approach."},{"text":"GPU access in WSL2 comes from the Windows-side driver through a passthrough layer, so installing a Linux driver inside the distribution replaces the working path with a broken one.","correct":true},{"text":"The CUDA build needs the full CUDA toolkit installed in the distribution, not just the driver, before it can see the device.","correct":false,"tempting":"Confuses compiling with running; pre-built binaries carry the runtime they need and do not require the toolkit."},{"text":"llama.cpp requires the Vulkan backend under WSL2 because CUDA kernels are unavailable through the passthrough interface.","correct":false,"tempting":"Invents a restriction; CUDA is the standard and best-supported path under WSL2."}]},"prev":{"slug":"interfaces","title":"Interfaces: what each one is for","url":"https://addaly.com/learn/running-models-yourself/interfaces"},"next":{"slug":"downloading-models-properly","title":"Downloading models without wasting a day","url":"https://addaly.com/learn/running-models-yourself/downloading-models-properly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Installing the right build for your hardware', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/installing-the-right-build (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Installing the right build for your hardware","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"The build has to match the accelerator, and a binary compiled without your backend runs silently on the CPU. Verify with a benchmark that names the device before configuring anything else.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-commonest-local-inference-bug","runs":[{"text":"The commonest local-inference bug"}]},{"t":"p","runs":[{"text":"A model runs, produces sensible text, and is thirty times slower than it should be. The cause, nearly always: the binary was compiled without support for your accelerator, so everything is on the CPU. "},{"text":"nvidia-smi","code":true},{"text":" shows 0% utilisation while the model is generating. Check that first, before any configuration."}]},{"t":"p","runs":[{"text":"The backend has to match your hardware, and there are five that matter."}]},{"t":"p","runs":[{"text":"CUDA","b":true},{"text":": NVIDIA. The best-supported path by a wide margin. Needs a driver new enough for the CUDA version the build was compiled against; the runtime ships with the build, so you do not need the full toolkit unless you are compiling."}]},{"t":"p","runs":[{"text":"ROCm","b":true},{"text":": AMD. Works well on supported cards, and the list of supported cards is shorter than the list of cards AMD sells. Check yours before planning around it. "},{"text":"HSA_OVERRIDE_GFX_VERSION","code":true},{"text":" is the environment variable that makes several unofficially supported cards work, and it is a well-trodden path with rough edges."}]},{"t":"p","runs":[{"text":"Metal","b":true},{"text":": Apple Silicon. Built in, no configuration, works."}]},{"t":"p","runs":[{"text":"Vulkan","b":true},{"text":": anything with a graphics driver, including Intel and AMD integrated graphics. Slower than the native backends and enormously faster than CPU-only for prompt processing. This is the backend most laptop owners should be using and most are not."}]},{"t":"p","runs":[{"text":"CPU","b":true},{"text":": always available, and the fallback everything silently lands on."}]},{"t":"h2","id":"getting-a-working-build-without-compiling","runs":[{"text":"Getting a working build without compiling"}]},{"t":"p","runs":[{"text":"The pre-built releases cover most people:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# Ollama, all platforms, picks its own backend\ncurl -fsSL https://ollama.com/install.sh | sh\n\n# llama.cpp on macOS\nbrew install llama.cpp\n\n# llama.cpp elsewhere: download a release archive matching your backend\n# from the project's releases page, e.g. -cuda-, -vulkan-, -hip-"},{"t":"p","runs":[{"text":"Verify immediately, before doing anything else:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-cli --version           # should name the backend\nllama-bench -m any-model.gguf -p 128 -n 32"},{"t":"p","runs":[{"text":"If "},{"text":"llama-bench","code":true},{"text":" reports a device that is not your GPU, stop and fix the build. Everything downstream is wasted effort until this is right."}]},{"t":"h2","id":"compiling-when-it-is-worth-it","runs":[{"text":"Compiling, when it is worth it"}]},{"t":"p","runs":[{"text":"Worth it for: a CPU with AVX-512 or AMX, an unusual AMD card, or a bleeding-edge model whose architecture is only in the main branch."}]},{"t":"code","lang":"bash","label":"Bash","text":"git clone https://github.com/ggml-org/llama.cpp\ncmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release\ncmake --build build --config Release -j"},{"t":"p","runs":[{"text":"Swap "},{"text":"-DGGML_CUDA=ON","code":true},{"text":" for "},{"text":"-DGGML_VULKAN=ON","code":true},{"text":", "},{"text":"-DGGML_HIP=ON","code":true},{"text":" or "},{"text":"-DGGML_METAL=ON","code":true},{"text":". Twenty minutes on a modern machine, and a native build can beat a generic one by a useful margin on prefill."}]},{"t":"h2","id":"python-environments-and-the-reason-to-isolate-them","runs":[{"text":"Python environments, and the reason to isolate them"}]},{"t":"p","runs":[{"text":"vLLM, transformers and the training tools bring large, version-sensitive dependency trees. Installing them into your system Python will eventually break something else you rely on. Use an isolated environment, and prefer a fast resolver:"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install uv\nuv venv --python 3.12 && source .venv/bin/activate\nuv pip install vllm"},{"t":"p","runs":[{"text":"vLLM in particular pins a specific PyTorch build against a specific CUDA version. Fighting that is a bad afternoon; a fresh environment per project is the cheap way out."}]},{"t":"h2","id":"docker-and-what-it-does-not-solve","runs":[{"text":"Docker, and what it does not solve"}]},{"t":"p","runs":[{"text":"Containers give you a known-good combination of driver-adjacent libraries, which removes most version arguments:"}]},{"t":"code","lang":"bash","label":"Bash","text":"docker run --gpus all -p 8000:8000 \\\n  -v ~/models:/models \\\n  vllm/vllm-openai:latest --model /models/my-model"},{"t":"p","runs":[{"text":"The host driver still has to be new enough: the container carries the runtime, not the kernel driver. And on Windows, GPU passthrough works through WSL2 with a Windows-side NVIDIA driver, not a driver installed inside the Linux distribution. Installing one there is a common and confusing mistake."}]},{"t":"h2","id":"a-working-order-that-avoids-most-trouble","runs":[{"text":"A working order that avoids most trouble"}]},{"t":"ol","start":1,"items":[[{"text":"Update the graphics driver."}],[{"text":"Install Ollama and run a small model. Confirm the GPU is used."}],[{"text":"Only then install anything more specialised, in its own environment."}],[{"text":"Record the versions that worked (driver, engine, CUDA or ROCm) in a text file."}]]},{"t":"p","runs":[{"text":"That last step sounds fussy until an upgrade breaks something and you need to know what changed. Engines in this field move weekly, and regressions are normal rather than exceptional."}]}],"check":{"question":[{"text":"On a Windows laptop with an NVIDIA GPU, someone installs a Linux NVIDIA driver inside their WSL2 distribution, then runs a CUDA build of llama.cpp there. It runs but shows no GPU use. What is wrong?"}],"options":[{"text":[{"text":"WSL2 cannot pass through a discrete GPU at all; CUDA workloads must run on Windows directly or in a virtual machine with device passthrough."}],"correct":false,"tempting":[{"text":"Denies a capability that exists and works well, leading to an unnecessary and expensive change of approach."}]},{"text":[{"text":"GPU access in WSL2 comes from the Windows-side driver through a passthrough layer, so installing a Linux driver inside the distribution replaces the working path with a broken one."}],"correct":true,"tempting":null},{"text":[{"text":"The CUDA build needs the full CUDA toolkit installed in the distribution, not just the driver, before it can see the device."}],"correct":false,"tempting":[{"text":"Confuses compiling with running; pre-built binaries carry the runtime they need and do not require the toolkit."}]},{"text":[{"text":"llama.cpp requires the Vulkan backend under WSL2 because CUDA kernels are unavailable through the passthrough interface."}],"correct":false,"tempting":[{"text":"Invents a restriction; CUDA is the standard and best-supported path under WSL2."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"interfaces","title":"Interfaces: what each one is for","minutes":6},"next":{"slug":"downloading-models-properly","title":"Downloading models without wasting a day","minutes":8}}}