{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"running-several-models","title":"One machine, several models","url":"https://addaly.com/learn/running-models-yourself/running-several-models","position":45,"of":82,"minutes":9,"answer":"Add the model sizes before designing anything: embedding and reranking models are small enough to stay resident, and several fine-tunes of one base should be served as adapters rather than as separate models.","keyIdea":"Add the model sizes before designing anything: embedding and reranking models are small enough to stay resident, and several fine-tunes of one base should be served as adapters rather than as separate models."},"body":{"format":"markdown","text":"## The problem\n\nA realistic system needs more than one model: a general assistant, a code model, an embedding model, a reranker, perhaps a transcription model. Their combined weights exceed your VRAM, and they are not all busy at once.\n\nFour arrangements exist, and the right one depends on how often each is used and how much latency a switch may cost.\n\n## 1. Keep everything resident\n\nSimplest, and correct when the total fits. An 8B at 4-bit is 4.9 GB, an embedding model 0.4 GB, a reranker 0.6 GB: under 6 GB together, comfortable on a 12 GB card with cache to spare.\n\nNote that the small models are small. People plan elaborate swapping schemes for components that would have fitted alongside everything else. Add the numbers first.\n\n## 2. Load on demand and unload when idle\n\nOllama does this by default: a model is loaded on first use and unloaded after five minutes idle. `OLLAMA_KEEP_ALIVE` controls the timeout and `OLLAMA_MAX_LOADED_MODELS` how many may be resident at once.\n\nThe cost is the load itself. Reading a 5 GB file from NVMe into VRAM is a few seconds; from a spinning disk or over a network share it can be a minute. That delay lands on the unlucky user whose request triggered the load.\n\nFor a proxy that does the same in front of llama.cpp, `llama-swap` sits between the client and several server configurations, starting and stopping them by the model name in the request. This gives you llama.cpp's full flag set with Ollama's convenience, which is a good combination.\n\n## 3. Several models, several servers, one router\n\nRun each model on its own port with its own tuned configuration, and put a router in front that dispatches by model name. LiteLLM's proxy is the common choice, and it adds per-key authentication, per-user rate limits, spend tracking and logging: the operational layer that inference engines deliberately do not provide.\n\nThis is the right shape once more than one person depends on the system. It also lets you mix local and hosted behind one endpoint, so a fallback to an API when the local model is unavailable is a routing rule rather than application code.\n\n## 4. One base model, many adapters\n\nIf your \"different models\" are actually fine-tunes of the same base, do not load several copies. LoRA adapters are small (tens of megabytes) and vLLM can serve many against one resident base model, selecting per request:\n\n```bash\nvllm serve <base-model> --enable-lora \\\n  --lora-modules legal=/adapters/legal support=/adapters/support\n```\n\nA request naming `legal` is served by the base weights plus that adapter. Memory cost is one base model plus a few megabytes each, instead of one full model each. For a team with several specialised variants this is the difference between one card and four.\n\n## The failure modes\n\n**Thrashing.** Two models alternating on a card too small for both, each evicting the other. Every request pays a load. Symptom: latency alternating between fast and terrible with no pattern in the requests. Fix by pinning the frequently used one resident and routing the other elsewhere, or by making one smaller.\n\n**Memory fragmentation over long uptimes.** Repeated load and unload cycles can leave a card unable to allocate a contiguous block it could have allocated at startup. Restarting the server clears it; a scheduled restart is a legitimate answer.\n\n**Silent fallback to CPU.** When VRAM is short, some stacks quietly place the new model in system memory instead of failing. It works and is thirty times slower. Check `nvidia-smi` after any change to the set of resident models.\n\n**Version drift.** Different models needing different engine versions is a real and unpleasant problem. Containers per model, behind one router, is the tidy solution.\n\n## Deciding\n\nAdd the sizes. If everything fits, keep it resident and stop thinking about it. If it does not, ask which model is on the hot path for user-facing latency: that one stays resident, and the rest load on demand or move to a second machine. Almost every arrangement people design is more complicated than this arithmetic requires."},"check":{"question":"A team serves four fine-tuned variants of the same 8B base model for four departments, loading and unloading them on a single 24 GB card, and sees loads triggered on most requests. What is the best fix?","options":[{"text":"Increase the keep-alive timeout so all four stay resident between requests.","correct":false,"tempting":"Four full 8B copies will not fit alongside their caches, so this converts thrashing into out-of-memory errors."},{"text":"Route each department to a fixed time window so only one variant is in use at a time.","correct":false,"tempting":"Solves the technical symptom by constraining the users, which is available in principle and unacceptable in practice."},{"text":"Serve one base model with the four fine-tunes as LoRA adapters selected per request, so only the base weights are resident.","correct":true},{"text":"Quantise all four to Q2 so that they fit simultaneously in 24 GB.","correct":false,"tempting":"Pays severe quality damage at 8B to solve a problem that adapters solve at no quality cost."}]},"prev":{"slug":"the-tuning-checklist","title":"The tuning checklist, in order","url":"https://addaly.com/learn/running-models-yourself/the-tuning-checklist"},"next":{"slug":"queueing-and-backpressure","title":"Queues, timeouts and refusing work gracefully","url":"https://addaly.com/learn/running-models-yourself/queueing-and-backpressure"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'One machine, several models', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/running-several-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"One machine, several models","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Add the model sizes before designing anything: embedding and reranking models are small enough to stay resident, and several fine-tunes of one base should be served as adapters rather than as separate models.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-problem","runs":[{"text":"The problem"}]},{"t":"p","runs":[{"text":"A realistic system needs more than one model: a general assistant, a code model, an embedding model, a reranker, perhaps a transcription model. Their combined weights exceed your VRAM, and they are not all busy at once."}]},{"t":"p","runs":[{"text":"Four arrangements exist, and the right one depends on how often each is used and how much latency a switch may cost."}]},{"t":"h2","id":"1-keep-everything-resident","runs":[{"text":"1. Keep everything resident"}]},{"t":"p","runs":[{"text":"Simplest, and correct when the total fits. An 8B at 4-bit is 4.9 GB, an embedding model 0.4 GB, a reranker 0.6 GB: under 6 GB together, comfortable on a 12 GB card with cache to spare."}]},{"t":"p","runs":[{"text":"Note that the small models are small. People plan elaborate swapping schemes for components that would have fitted alongside everything else. Add the numbers first."}]},{"t":"h2","id":"2-load-on-demand-and-unload-when-idle","runs":[{"text":"2. Load on demand and unload when idle"}]},{"t":"p","runs":[{"text":"Ollama does this by default: a model is loaded on first use and unloaded after five minutes idle. "},{"text":"OLLAMA_KEEP_ALIVE","code":true},{"text":" controls the timeout and "},{"text":"OLLAMA_MAX_LOADED_MODELS","code":true},{"text":" how many may be resident at once."}]},{"t":"p","runs":[{"text":"The cost is the load itself. Reading a 5 GB file from NVMe into VRAM is a few seconds; from a spinning disk or over a network share it can be a minute. That delay lands on the unlucky user whose request triggered the load."}]},{"t":"p","runs":[{"text":"For a proxy that does the same in front of llama.cpp, "},{"text":"llama-swap","code":true},{"text":" sits between the client and several server configurations, starting and stopping them by the model name in the request. This gives you llama.cpp’s full flag set with Ollama’s convenience, which is a good combination."}]},{"t":"h2","id":"3-several-models-several-servers-one-router","runs":[{"text":"3. Several models, several servers, one router"}]},{"t":"p","runs":[{"text":"Run each model on its own port with its own tuned configuration, and put a router in front that dispatches by model name. LiteLLM’s proxy is the common choice, and it adds per-key authentication, per-user rate limits, spend tracking and logging: the operational layer that inference engines deliberately do not provide."}]},{"t":"p","runs":[{"text":"This is the right shape once more than one person depends on the system. It also lets you mix local and hosted behind one endpoint, so a fallback to an API when the local model is unavailable is a routing rule rather than application code."}]},{"t":"h2","id":"4-one-base-model-many-adapters","runs":[{"text":"4. One base model, many adapters"}]},{"t":"p","runs":[{"text":"If your “different models” are actually fine-tunes of the same base, do not load several copies. LoRA adapters are small (tens of megabytes) and vLLM can serve many against one resident base model, selecting per request:"}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve <base-model> --enable-lora \\\n  --lora-modules legal=/adapters/legal support=/adapters/support"},{"t":"p","runs":[{"text":"A request naming "},{"text":"legal","code":true},{"text":" is served by the base weights plus that adapter. Memory cost is one base model plus a few megabytes each, instead of one full model each. For a team with several specialised variants this is the difference between one card and four."}]},{"t":"h2","id":"the-failure-modes","runs":[{"text":"The failure modes"}]},{"t":"p","runs":[{"text":"Thrashing.","b":true},{"text":" Two models alternating on a card too small for both, each evicting the other. Every request pays a load. Symptom: latency alternating between fast and terrible with no pattern in the requests. Fix by pinning the frequently used one resident and routing the other elsewhere, or by making one smaller."}]},{"t":"p","runs":[{"text":"Memory fragmentation over long uptimes.","b":true},{"text":" Repeated load and unload cycles can leave a card unable to allocate a contiguous block it could have allocated at startup. Restarting the server clears it; a scheduled restart is a legitimate answer."}]},{"t":"p","runs":[{"text":"Silent fallback to CPU.","b":true},{"text":" When VRAM is short, some stacks quietly place the new model in system memory instead of failing. It works and is thirty times slower. Check "},{"text":"nvidia-smi","code":true},{"text":" after any change to the set of resident models."}]},{"t":"p","runs":[{"text":"Version drift.","b":true},{"text":" Different models needing different engine versions is a real and unpleasant problem. Containers per model, behind one router, is the tidy solution."}]},{"t":"h2","id":"deciding","runs":[{"text":"Deciding"}]},{"t":"p","runs":[{"text":"Add the sizes. If everything fits, keep it resident and stop thinking about it. If it does not, ask which model is on the hot path for user-facing latency: that one stays resident, and the rest load on demand or move to a second machine. Almost every arrangement people design is more complicated than this arithmetic requires."}]}],"check":{"question":[{"text":"A team serves four fine-tuned variants of the same 8B base model for four departments, loading and unloading them on a single 24 GB card, and sees loads triggered on most requests. What is the best fix?"}],"options":[{"text":[{"text":"Increase the keep-alive timeout so all four stay resident between requests."}],"correct":false,"tempting":[{"text":"Four full 8B copies will not fit alongside their caches, so this converts thrashing into out-of-memory errors."}]},{"text":[{"text":"Route each department to a fixed time window so only one variant is in use at a time."}],"correct":false,"tempting":[{"text":"Solves the technical symptom by constraining the users, which is available in principle and unacceptable in practice."}]},{"text":[{"text":"Serve one base model with the four fine-tunes as LoRA adapters selected per request, so only the base weights are resident."}],"correct":true,"tempting":null},{"text":[{"text":"Quantise all four to Q2 so that they fit simultaneously in 24 GB."}],"correct":false,"tempting":[{"text":"Pays severe quality damage at 8B to solve a problem that adapters solve at no quality cost."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"the-tuning-checklist","title":"The tuning checklist, in order","minutes":8},"next":{"slug":"queueing-and-backpressure","title":"Queues, timeouts and refusing work gracefully","minutes":8}}}