{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"running-it-as-a-service","title":"Running it as a service that survives a reboot","url":"https://addaly.com/learn/running-models-yourself/running-it-as-a-service","position":60,"of":82,"minutes":8,"answer":"A supervisor turns a terminal process into a service, but the details that matter are a long start timeout so a large model can finish loading.","keyIdea":"A supervisor turns a terminal process into a service, but the details that matter are a long start timeout so a large model can finish loading, a warm-up request so the first user does not pay for disk reads, and a look at the kernel log whenever it restarts, because an out-of-memory kill leaves no trace in the engine's own logs."},"body":{"format":"markdown","text":"## The gap between working and running\n\nA model started in a terminal stops when the terminal closes, when the SSH session drops, when the machine reboots after a kernel update, and when the process runs out of memory. A service supervisor closes all four gaps. On Linux that is `systemd`, which is already installed; on macOS it is `launchd`; on a Windows machine the practical answer is to run the engine inside WSL2 and supervise it there.\n\nA minimal unit for a llama.cpp server:\n\n```ini\n# /etc/systemd/system/llama.service\n[Unit]\nDescription=llama.cpp server\nAfter=network-online.target\n\n[Service]\nUser=llama\nGroup=llama\nExecStart=/opt/llama.cpp/llama-server \\\n  -m /srv/models/qwen2.5-7b-instruct-q4_k_m.gguf \\\n  --host 127.0.0.1 --port 8080 -c 8192 -ngl 99\nRestart=always\nRestartSec=5\nTimeoutStartSec=600\n\n[Install]\nWantedBy=multi-user.target\n```\n\n```bash\nsudo systemctl daemon-reload && sudo systemctl enable --now llama\njournalctl -u llama -f      # the logs, with no extra software\n```\n\nFour things in that file earn their place. `User=llama` means a compromise of the engine is not a compromise of your account: create the user with `useradd -r -s /usr/sbin/nologin llama` and give it read access to the model directory and nothing else. `Restart=always` covers the crash case. `RestartSec=5` prevents a crash loop from hammering the disk. And `TimeoutStartSec=600` matters more than it looks: loading a 40GB model from a spinning disk or over a network mount can take minutes, and the default 90-second start timeout will kill it mid-load and then restart it forever.\n\n## Cold start is the number users feel\n\nThe first request after a restart pays for reading the weights from disk into memory. On an NVMe drive at 3GB/s a 20GB model takes about seven seconds; on a SATA SSD at 500MB/s the same model takes forty. The second request is instant because the file is in the page cache, which is why the problem is invisible to whoever tested it twice.\n\nOllama adds its own wrinkle. It unloads a model after five minutes idle by default, so a service used a few times an hour reloads constantly. `OLLAMA_KEEP_ALIVE=-1` keeps it resident indefinitely; `OLLAMA_KEEP_ALIVE=30m` is the sensible middle. The cost is honest: a resident model holds its VRAM whether or not anybody is using it, which is exactly the idle-cost problem the economics module works through.\n\nTo remove cold start entirely, send a warming request in the unit itself:\n\n```ini\nExecStartPost=/bin/sh -c 'until curl -sf -m 2 http://127.0.0.1:8080/health; do sleep 2; done; \\\n  curl -s http://127.0.0.1:8080/v1/chat/completions -H \"Content-Type: application/json\" \\\n  -d \"{\\\"model\\\":\\\"local\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"hi\\\"}],\\\"max_tokens\\\":1}\" >/dev/null'\n```\n\nThe service then reports itself started only once it can actually answer, which is what a dependent service or a health check wants to know.\n\n## The out-of-memory kill nobody sees\n\nThe most confusing production failure is a process that disappears with no error in its own logs. On Linux that is the kernel's OOM killer, and it leaves its evidence elsewhere:\n\n```bash\njournalctl -k | grep -i 'killed process'\n# Out of memory: Killed process 4127 (llama-server) total-vm:...\n```\n\nThis is system RAM, not VRAM. It happens when the KV cache spills to host memory, when a second model is loaded alongside the first, or when a memory-mapped model plus a large batch exceeds what is free. `Restart=always` brings it back, which hides the cause; so check the kernel log whenever a service has restarted for no apparent reason. If it recurs, either cap the context, reduce concurrency, or add swap as a shock absorber, knowing that a model paging to swap is effectively stopped rather than slow.\n\n## What to monitor, minimally\n\nThree numbers tell you almost everything, and all three are free. `nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv -l 5` prints them once every five seconds. Utilisation near zero while users complain means the bottleneck is elsewhere: the gateway, the network, or a queue. Memory climbing steadily over hours means a cache is not being released. Temperature at the card's limit means the fans have lost and the clock has already been reduced, which shows up as a gradual, unexplained loss of tokens per second rather than an error.\n\n## Two host settings that cause phantom failures\n\nTwo things outside the unit file produce failures that look like the engine and are not.\n\nThe first is the model path. If the weights live on a network share or a second disk, `After=network-online.target` is not enough: the unit needs `RequiresMountsFor=/srv/models` so that it waits for the filesystem instead of starting, failing to open the file, and restarting in a loop that appears to blame the model. Adding `ExecStartPre=/usr/bin/test -r /srv/models/model.gguf` makes the journal say so on the first attempt rather than the twentieth.\n\nThe second is GPU persistence mode. With it off, the driver tears down its state whenever no process is holding the GPU, and the next start pays several seconds to reinitialise it. `sudo nvidia-smi -pm 1` turns it on, and the `nvidia-persistenced` daemon keeps it on across reboots. On a machine where the engine restarts occasionally, this removes a delay that is otherwise blamed on the model loading.\n\n> A service that restarts silently and a service that never fails look identical on a dashboard. Only the restart counter tells them apart."},"check":{"question":"A systemd-managed engine loading a 40GB model enters a loop: it starts, is killed after about ninety seconds, and starts again. The engine's own log shows no error. What is the most likely cause?","options":[{"text":"systemd's ninety-second default start timeout fires while the weights are still loading.","correct":true},{"text":"Restart=always is creating the loop and should be changed to Restart=on-failure.","correct":false,"tempting":"Changes what happens after the kill without addressing why the kill happens; the restart policy is correct and the timeout is not."},{"text":"The model file is corrupt, so loading never completes and systemd gives up.","correct":false,"tempting":"A corrupt file produces an explicit parse or checksum error in the engine's log rather than a silent kill on a suspiciously round interval."},{"text":"The GPU has insufficient VRAM for the model, and the driver is terminating the allocation.","correct":false,"tempting":"A VRAM shortfall produces an explicit out-of-memory error from the engine at allocation time, and it would not be tied to a fixed ninety-second mark."}]},"prev":{"slug":"containers-and-driver-versions","title":"Containers, drivers and the version matrix","url":"https://addaly.com/learn/running-models-yourself/containers-and-driver-versions"},"next":{"slug":"logging-without-leaking","title":"Logging without building a liability","url":"https://addaly.com/learn/running-models-yourself/logging-without-leaking"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Running it as a service that survives a reboot', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/running-it-as-a-service (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Running it as a service that survives a reboot","courseTitle":"Running Models Yourself","module":{"index":7,"slug":"serving-it-to-others","title":"Serving it to other people"},"answer":null,"keyIdea":"A supervisor turns a terminal process into a service, but the details that matter are a long start timeout so a large model can finish loading, a warm-up request so the first user does not pay for disk reads, and a look at the kernel log whenever it restarts, because an out-of-memory kill leaves no trace in the engine’s own logs.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-gap-between-working-and-running","runs":[{"text":"The gap between working and running"}]},{"t":"p","runs":[{"text":"A model started in a terminal stops when the terminal closes, when the SSH session drops, when the machine reboots after a kernel update, and when the process runs out of memory. A service supervisor closes all four gaps. On Linux that is "},{"text":"systemd","code":true},{"text":", which is already installed; on macOS it is "},{"text":"launchd","code":true},{"text":"; on a Windows machine the practical answer is to run the engine inside WSL2 and supervise it there."}]},{"t":"p","runs":[{"text":"A minimal unit for a llama.cpp server:"}]},{"t":"code","lang":"ini","label":"INI","text":"# /etc/systemd/system/llama.service\n[Unit]\nDescription=llama.cpp server\nAfter=network-online.target\n\n[Service]\nUser=llama\nGroup=llama\nExecStart=/opt/llama.cpp/llama-server \\\n  -m /srv/models/qwen2.5-7b-instruct-q4_k_m.gguf \\\n  --host 127.0.0.1 --port 8080 -c 8192 -ngl 99\nRestart=always\nRestartSec=5\nTimeoutStartSec=600\n\n[Install]\nWantedBy=multi-user.target"},{"t":"code","lang":"bash","label":"Bash","text":"sudo systemctl daemon-reload && sudo systemctl enable --now llama\njournalctl -u llama -f      # the logs, with no extra software"},{"t":"p","runs":[{"text":"Four things in that file earn their place. "},{"text":"User=llama","code":true},{"text":" means a compromise of the engine is not a compromise of your account: create the user with "},{"text":"useradd -r -s /usr/sbin/nologin llama","code":true},{"text":" and give it read access to the model directory and nothing else. "},{"text":"Restart=always","code":true},{"text":" covers the crash case. "},{"text":"RestartSec=5","code":true},{"text":" prevents a crash loop from hammering the disk. And "},{"text":"TimeoutStartSec=600","code":true},{"text":" matters more than it looks: loading a 40GB model from a spinning disk or over a network mount can take minutes, and the default 90-second start timeout will kill it mid-load and then restart it forever."}]},{"t":"h2","id":"cold-start-is-the-number-users-feel","runs":[{"text":"Cold start is the number users feel"}]},{"t":"p","runs":[{"text":"The first request after a restart pays for reading the weights from disk into memory. On an NVMe drive at 3GB/s a 20GB model takes about seven seconds; on a SATA SSD at 500MB/s the same model takes forty. The second request is instant because the file is in the page cache, which is why the problem is invisible to whoever tested it twice."}]},{"t":"p","runs":[{"text":"Ollama adds its own wrinkle. It unloads a model after five minutes idle by default, so a service used a few times an hour reloads constantly. "},{"text":"OLLAMA_KEEP_ALIVE=-1","code":true},{"text":" keeps it resident indefinitely; "},{"text":"OLLAMA_KEEP_ALIVE=30m","code":true},{"text":" is the sensible middle. The cost is honest: a resident model holds its VRAM whether or not anybody is using it, which is exactly the idle-cost problem the economics module works through."}]},{"t":"p","runs":[{"text":"To remove cold start entirely, send a warming request in the unit itself:"}]},{"t":"code","lang":"ini","label":"INI","text":"ExecStartPost=/bin/sh -c 'until curl -sf -m 2 http://127.0.0.1:8080/health; do sleep 2; done; \\\n  curl -s http://127.0.0.1:8080/v1/chat/completions -H \"Content-Type: application/json\" \\\n  -d \"{\\\"model\\\":\\\"local\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"hi\\\"}],\\\"max_tokens\\\":1}\" >/dev/null'"},{"t":"p","runs":[{"text":"The service then reports itself started only once it can actually answer, which is what a dependent service or a health check wants to know."}]},{"t":"h2","id":"the-out-of-memory-kill-nobody-sees","runs":[{"text":"The out-of-memory kill nobody sees"}]},{"t":"p","runs":[{"text":"The most confusing production failure is a process that disappears with no error in its own logs. On Linux that is the kernel’s OOM killer, and it leaves its evidence elsewhere:"}]},{"t":"code","lang":"bash","label":"Bash","text":"journalctl -k | grep -i 'killed process'\n# Out of memory: Killed process 4127 (llama-server) total-vm:..."},{"t":"p","runs":[{"text":"This is system RAM, not VRAM. It happens when the KV cache spills to host memory, when a second model is loaded alongside the first, or when a memory-mapped model plus a large batch exceeds what is free. "},{"text":"Restart=always","code":true},{"text":" brings it back, which hides the cause; so check the kernel log whenever a service has restarted for no apparent reason. If it recurs, either cap the context, reduce concurrency, or add swap as a shock absorber, knowing that a model paging to swap is effectively stopped rather than slow."}]},{"t":"h2","id":"what-to-monitor-minimally","runs":[{"text":"What to monitor, minimally"}]},{"t":"p","runs":[{"text":"Three numbers tell you almost everything, and all three are free. "},{"text":"nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv -l 5","code":true},{"text":" prints them once every five seconds. Utilisation near zero while users complain means the bottleneck is elsewhere: the gateway, the network, or a queue. Memory climbing steadily over hours means a cache is not being released. Temperature at the card’s limit means the fans have lost and the clock has already been reduced, which shows up as a gradual, unexplained loss of tokens per second rather than an error."}]},{"t":"h2","id":"two-host-settings-that-cause-phantom-failures","runs":[{"text":"Two host settings that cause phantom failures"}]},{"t":"p","runs":[{"text":"Two things outside the unit file produce failures that look like the engine and are not."}]},{"t":"p","runs":[{"text":"The first is the model path. If the weights live on a network share or a second disk, "},{"text":"After=network-online.target","code":true},{"text":" is not enough: the unit needs "},{"text":"RequiresMountsFor=/srv/models","code":true},{"text":" so that it waits for the filesystem instead of starting, failing to open the file, and restarting in a loop that appears to blame the model. Adding "},{"text":"ExecStartPre=/usr/bin/test -r /srv/models/model.gguf","code":true},{"text":" makes the journal say so on the first attempt rather than the twentieth."}]},{"t":"p","runs":[{"text":"The second is GPU persistence mode. With it off, the driver tears down its state whenever no process is holding the GPU, and the next start pays several seconds to reinitialise it. "},{"text":"sudo nvidia-smi -pm 1","code":true},{"text":" turns it on, and the "},{"text":"nvidia-persistenced","code":true},{"text":" daemon keeps it on across reboots. On a machine where the engine restarts occasionally, this removes a delay that is otherwise blamed on the model loading."}]},{"t":"quote","runs":[{"text":"A service that restarts silently and a service that never fails look identical on a dashboard. Only the restart counter tells them apart."}]}],"check":{"question":[{"text":"A systemd-managed engine loading a 40GB model enters a loop: it starts, is killed after about ninety seconds, and starts again. The engine’s own log shows no error. What is the most likely cause?"}],"options":[{"text":[{"text":"systemd’s ninety-second default start timeout fires while the weights are still loading."}],"correct":true,"tempting":null},{"text":[{"text":"Restart=always is creating the loop and should be changed to Restart=on-failure."}],"correct":false,"tempting":[{"text":"Changes what happens after the kill without addressing why the kill happens; the restart policy is correct and the timeout is not."}]},{"text":[{"text":"The model file is corrupt, so loading never completes and systemd gives up."}],"correct":false,"tempting":[{"text":"A corrupt file produces an explicit parse or checksum error in the engine’s log rather than a silent kill on a suspiciously round interval."}]},{"text":[{"text":"The GPU has insufficient VRAM for the model, and the driver is terminating the allocation."}],"correct":false,"tempting":[{"text":"A VRAM shortfall produces an explicit out-of-memory error from the engine at allocation time, and it would not be tied to a fixed ninety-second mark."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"containers-and-driver-versions","title":"Containers, drivers and the version matrix","minutes":9},"next":{"slug":"logging-without-leaking","title":"Logging without building a liability","minutes":9}}}