Installing the right build for your hardware

The commonest local-inference bug

A model runs, produces sensible text, and is thirty times slower than it should be. The cause, nearly always: the binary was compiled without support for your accelerator, so everything is on the CPU. nvidia-smi shows 0% utilisation while the model is generating. Check that first, before any configuration.

The backend has to match your hardware, and there are five that matter.

CUDA: NVIDIA. The best-supported path by a wide margin. Needs a driver new enough for the CUDA version the build was compiled against; the runtime ships with the build, so you do not need the full toolkit unless you are compiling.

ROCm: AMD. Works well on supported cards, and the list of supported cards is shorter than the list of cards AMD sells. Check yours before planning around it. HSA_OVERRIDE_GFX_VERSION is the environment variable that makes several unofficially supported cards work, and it is a well-trodden path with rough edges.

Metal: Apple Silicon. Built in, no configuration, works.

Vulkan: anything with a graphics driver, including Intel and AMD integrated graphics. Slower than the native backends and enormously faster than CPU-only for prompt processing. This is the backend most laptop owners should be using and most are not.

CPU: always available, and the fallback everything silently lands on.

Getting a working build without compiling

The pre-built releases cover most people:

# Ollama, all platforms, picks its own backend
curl -fsSL https://ollama.com/install.sh | sh

# llama.cpp on macOS
brew install llama.cpp

# llama.cpp elsewhere: download a release archive matching your backend
# from the project's releases page, e.g. -cuda-, -vulkan-, -hip-

Verify immediately, before doing anything else:

llama-cli --version           # should name the backend
llama-bench -m any-model.gguf -p 128 -n 32

If llama-bench reports a device that is not your GPU, stop and fix the build. Everything downstream is wasted effort until this is right.

Compiling, when it is worth it

Worth it for: a CPU with AVX-512 or AMX, an unusual AMD card, or a bleeding-edge model whose architecture is only in the main branch.

git clone https://github.com/ggml-org/llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Swap -DGGML_CUDA=ON for -DGGML_VULKAN=ON, -DGGML_HIP=ON or -DGGML_METAL=ON. Twenty minutes on a modern machine, and a native build can beat a generic one by a useful margin on prefill.

Python environments, and the reason to isolate them

vLLM, transformers and the training tools bring large, version-sensitive dependency trees. Installing them into your system Python will eventually break something else you rely on. Use an isolated environment, and prefer a fast resolver:

pip install uv
uv venv --python 3.12 && source .venv/bin/activate
uv pip install vllm

vLLM in particular pins a specific PyTorch build against a specific CUDA version. Fighting that is a bad afternoon; a fresh environment per project is the cheap way out.

Docker, and what it does not solve

Containers give you a known-good combination of driver-adjacent libraries, which removes most version arguments:

docker run --gpus all -p 8000:8000 \
  -v ~/models:/models \
  vllm/vllm-openai:latest --model /models/my-model

The host driver still has to be new enough: the container carries the runtime, not the kernel driver. And on Windows, GPU passthrough works through WSL2 with a Windows-side NVIDIA driver, not a driver installed inside the Linux distribution. Installing one there is a common and confusing mistake.

A working order that avoids most trouble

1. Update the graphics driver.
2. Install Ollama and run a small model. Confirm the GPU is used.
3. Only then install anything more specialised, in its own environment.
4. Record the versions that worked (driver, engine, CUDA or ROCm) in a text file.

That last step sounds fussy until an upgrade breaks something and you need to know what changed. Engines in this field move weekly, and regressions are normal rather than exceptional.