One machine, several models

The problem

A realistic system needs more than one model: a general assistant, a code model, an embedding model, a reranker, perhaps a transcription model. Their combined weights exceed your VRAM, and they are not all busy at once.

Four arrangements exist, and the right one depends on how often each is used and how much latency a switch may cost.

1. Keep everything resident

Simplest, and correct when the total fits. An 8B at 4-bit is 4.9 GB, an embedding model 0.4 GB, a reranker 0.6 GB: under 6 GB together, comfortable on a 12 GB card with cache to spare.

Note that the small models are small. People plan elaborate swapping schemes for components that would have fitted alongside everything else. Add the numbers first.

2. Load on demand and unload when idle

Ollama does this by default: a model is loaded on first use and unloaded after five minutes idle. OLLAMA_KEEP_ALIVE controls the timeout and OLLAMA_MAX_LOADED_MODELS how many may be resident at once.

The cost is the load itself. Reading a 5 GB file from NVMe into VRAM is a few seconds; from a spinning disk or over a network share it can be a minute. That delay lands on the unlucky user whose request triggered the load.

For a proxy that does the same in front of llama.cpp, llama-swap sits between the client and several server configurations, starting and stopping them by the model name in the request. This gives you llama.cpp’s full flag set with Ollama’s convenience, which is a good combination.

3. Several models, several servers, one router

Run each model on its own port with its own tuned configuration, and put a router in front that dispatches by model name. LiteLLM’s proxy is the common choice, and it adds per-key authentication, per-user rate limits, spend tracking and logging: the operational layer that inference engines deliberately do not provide.

This is the right shape once more than one person depends on the system. It also lets you mix local and hosted behind one endpoint, so a fallback to an API when the local model is unavailable is a routing rule rather than application code.

4. One base model, many adapters

If your “different models” are actually fine-tunes of the same base, do not load several copies. LoRA adapters are small (tens of megabytes) and vLLM can serve many against one resident base model, selecting per request:

vllm serve <base-model> --enable-lora \
  --lora-modules legal=/adapters/legal support=/adapters/support

A request naming legal is served by the base weights plus that adapter. Memory cost is one base model plus a few megabytes each, instead of one full model each. For a team with several specialised variants this is the difference between one card and four.

The failure modes

Thrashing. Two models alternating on a card too small for both, each evicting the other. Every request pays a load. Symptom: latency alternating between fast and terrible with no pattern in the requests. Fix by pinning the frequently used one resident and routing the other elsewhere, or by making one smaller.

Memory fragmentation over long uptimes. Repeated load and unload cycles can leave a card unable to allocate a contiguous block it could have allocated at startup. Restarting the server clears it; a scheduled restart is a legitimate answer.

Silent fallback to CPU. When VRAM is short, some stacks quietly place the new model in system memory instead of failing. It works and is thirty times slower. Check nvidia-smi after any change to the set of resident models.

Version drift. Different models needing different engine versions is a real and unpleasant problem. Containers per model, behind one router, is the tidy solution.

Deciding

Add the sizes. If everything fits, keep it resident and stop thinking about it. If it does not, ask which model is on the hot path for user-facing latency: that one stays resident, and the rest load on demand or move to a second machine. Almost every arrangement people design is more complicated than this arithmetic requires.