Running it as a service that survives a reboot

The gap between working and running

A model started in a terminal stops when the terminal closes, when the SSH session drops, when the machine reboots after a kernel update, and when the process runs out of memory. A service supervisor closes all four gaps. On Linux that is systemd, which is already installed; on macOS it is launchd; on a Windows machine the practical answer is to run the engine inside WSL2 and supervise it there.

A minimal unit for a llama.cpp server:

# /etc/systemd/system/llama.service
[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
User=llama
Group=llama
ExecStart=/opt/llama.cpp/llama-server \
  -m /srv/models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 -c 8192 -ngl 99
Restart=always
RestartSec=5
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now llama
journalctl -u llama -f      # the logs, with no extra software

Four things in that file earn their place. User=llama means a compromise of the engine is not a compromise of your account: create the user with useradd -r -s /usr/sbin/nologin llama and give it read access to the model directory and nothing else. Restart=always covers the crash case. RestartSec=5 prevents a crash loop from hammering the disk. And TimeoutStartSec=600 matters more than it looks: loading a 40GB model from a spinning disk or over a network mount can take minutes, and the default 90-second start timeout will kill it mid-load and then restart it forever.

Cold start is the number users feel

The first request after a restart pays for reading the weights from disk into memory. On an NVMe drive at 3GB/s a 20GB model takes about seven seconds; on a SATA SSD at 500MB/s the same model takes forty. The second request is instant because the file is in the page cache, which is why the problem is invisible to whoever tested it twice.

Ollama adds its own wrinkle. It unloads a model after five minutes idle by default, so a service used a few times an hour reloads constantly. OLLAMA_KEEP_ALIVE=-1 keeps it resident indefinitely; OLLAMA_KEEP_ALIVE=30m is the sensible middle. The cost is honest: a resident model holds its VRAM whether or not anybody is using it, which is exactly the idle-cost problem the economics module works through.

To remove cold start entirely, send a warming request in the unit itself:

ExecStartPost=/bin/sh -c 'until curl -sf -m 2 http://127.0.0.1:8080/health; do sleep 2; done; \
  curl -s http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d "{\"model\":\"local\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":1}" >/dev/null'

The service then reports itself started only once it can actually answer, which is what a dependent service or a health check wants to know.

The out-of-memory kill nobody sees

The most confusing production failure is a process that disappears with no error in its own logs. On Linux that is the kernel’s OOM killer, and it leaves its evidence elsewhere:

journalctl -k | grep -i 'killed process'
# Out of memory: Killed process 4127 (llama-server) total-vm:...

This is system RAM, not VRAM. It happens when the KV cache spills to host memory, when a second model is loaded alongside the first, or when a memory-mapped model plus a large batch exceeds what is free. Restart=always brings it back, which hides the cause; so check the kernel log whenever a service has restarted for no apparent reason. If it recurs, either cap the context, reduce concurrency, or add swap as a shock absorber, knowing that a model paging to swap is effectively stopped rather than slow.

What to monitor, minimally

Three numbers tell you almost everything, and all three are free. nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv -l 5 prints them once every five seconds. Utilisation near zero while users complain means the bottleneck is elsewhere: the gateway, the network, or a queue. Memory climbing steadily over hours means a cache is not being released. Temperature at the card’s limit means the fans have lost and the clock has already been reduced, which shows up as a gradual, unexplained loss of tokens per second rather than an error.

Two host settings that cause phantom failures

Two things outside the unit file produce failures that look like the engine and are not.

The first is the model path. If the weights live on a network share or a second disk, After=network-online.target is not enough: the unit needs RequiresMountsFor=/srv/models so that it waits for the filesystem instead of starting, failing to open the file, and restarting in a loop that appears to blame the model. Adding ExecStartPre=/usr/bin/test -r /srv/models/model.gguf makes the journal say so on the first attempt rather than the twentieth.

The second is GPU persistence mode. With it off, the driver tears down its state whenever no process is holding the GPU, and the next start pays several seconds to reinitialise it. sudo nvidia-smi -pm 1 turns it on, and the nvidia-persistenced daemon keeps it on across reboots. On a machine where the engine restarts occasionally, this removes a delay that is otherwise blamed on the model loading.

A service that restarts silently and a service that never fails look identical on a dashboard. Only the restart counter tells them apart.