Serving many adapters from one base

The arithmetic that makes this interesting

Fifty customers, each wanting a model tuned on their data.

• Fifty merged models, 8B each at 16 GB: 800 GB of weights and fifty serving processes.
• One base plus fifty adapters: 16 GB plus fifty times 160 MB, so about 24 GB. One process.

That is the whole argument, and it is decisive. Multi-adapter serving turns per-customer models from an impossible proposition into a routine one.

How it works

The base weights stay resident. For each request, the server applies the named adapter’s low-rank matrices during the forward pass. Modern implementations batch requests using different adapters together, computing the base matrix multiplication once for the whole batch and the small per-adapter corrections separately.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora --max-loras 8 --max-lora-rank 32 \
  --lora-modules acme=/adapters/acme globex=/adapters/globex

Then the request names the adapter as its model:

{"model": "acme", "messages": [...]}

LoRAX is a dedicated server built around the same idea with dynamic loading from a registry; vLLM’s support is built in and sufficient for most cases.

The costs

Throughput. The extra low-rank multiplications cost throughput against a merged model. The cost grows with rank and with the number of distinct adapters in a batch, and published measurements run from small to about half. Measure it on your own traffic before relying on a number. Usually worth paying for the operational simplification.

Concurrency limits. --max-loras caps how many adapters are active in the GPU at once. Beyond it, adapters are swapped in from host memory or disk, adding latency to the first request that needs one. If you have 500 adapters and 20 are hot, size the cap for the hot set.

Uniformity requirements. Every adapter must share the base model, and rank must be at or below --max-lora-rank. Plan this at training time: pick one rank for the whole fleet, because a single rank-128 adapter raises the ceiling and the memory cost for all of them.

Cold start. Loading an adapter from object storage on first use costs hundreds of milliseconds. Prewarm the ones you know will be needed.

When to merge instead

Merge when there is one adapter and it is always used. You get the base model’s exact serving performance, any inference engine works without adapter support, and you can quantise the result freely.

Keep adapters separate when you have several, when you want to switch without a redeploy, or when you want to A/B two versions against the same base, which is a useful capability that merging removes.

Composing adapters

The libraries let you load several adapters at once and combine them, with weights:

model.add_weighted_adapter(["formal", "legal"], [0.6, 0.4], "combined")

It works in the sense that it runs and produces something. Whether it produces the intended combination is much less certain: adapters trained independently occupy overlapping subspaces, and adding them can interfere in ways that degrade both behaviours rather than combining them. There are methods designed for this (concatenation-based merging, and orthogonality-constrained training) and the results are mixed.

Treat composition as an experiment, not a feature. If you need two behaviours together, train one adapter on both datasets. That reliably works, and it costs one more run.

Evaluating a fleet

Fifty adapters is fifty models, and each needs its own held-out set, its own canary and its own number. That is the cost people discover after building the serving layer.

The workable approach is a shared harness with per-tenant data: one evaluation script, one command, a results file per adapter, and a single summary that flags any tenant whose metric has fallen since the last build. Without that summary nobody looks at forty-nine of the fifty, and a regression affecting one customer surfaces as a support ticket.

Budget for this before you commit to per-customer adapters. It is usually a larger ongoing cost than the serving.

The per-customer question nobody asks first

Before building this, check whether you need it. One adapter trained on all customers’ data, with the customer identified in the system prompt, frequently matches per-customer adapters and is enormously simpler: one artefact, one evaluation, one thing to retrain when the base model moves.

Per-customer adapters earn their complexity when the customers differ in ways a prompt cannot express, or when the data may not be mixed for contractual or regulatory reasons. That second reason is often the real one, and it is a good reason. But it is worth knowing that you are paying for isolation rather than for quality.