Serving what you made, merged or not

Two ways to deploy, with different properties

An adapter can stay separate or be folded into the weights. The choice is not about quality; it is about how many adapters you have and where they need to run.

Keep it separate. The base model sits in VRAM once, and adapters are applied at runtime. vLLM supports this directly:

Bash

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-lora --max-loras 4 --max-lora-rank 32 \
  --lora-modules triage=/srv/adapters/triage support=/srv/adapters/support

Callers then ask for triage or support as if they were separate models, and both are served from one copy of the base. For a team with several task adapters this is a large memory saving: one 14GB base and three 40MB adapters instead of three 14GB models.

Merge it. Fold the update into the weights and produce a single ordinary model:

Python

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "out/checkpoint-150").merge_and_unload()
merged.save_pretrained("merged-model"); tok.save_pretrained("merged-model")

The arithmetic is exactly the formula from earlier: each adapted matrix becomes W + (alpha/r) * B @ A, computed once. The result is a normal model that any engine can load, that can be converted to GGUF and quantised, and that carries no per-token adapter overhead. This is the right choice when one adapter is the product, when it must run in llama.cpp or Ollama, or when it has to work on a machine with no adapter support at all.

The cost of keeping them separate

Multi-adapter serving is not free. Each token’s computation includes the low-rank product, which is a small overhead per request. More significantly, a batch containing requests for several different adapters is less efficient than a batch for one, because the adapter work cannot be shared across them. With a handful of adapters and modest traffic this is invisible; with dozens of adapters and heavy concurrency it is measurable.

There is also a hard constraint people meet late: --max-lora-rank must be at least the rank of every adapter you intend to load. An adapter trained at rank 64 will not load on a server configured for 16, and the error arrives when someone tries to use it rather than at startup.

Merging after QLoRA, again

The caution from the QLoRA lesson lands here. If the adapter was trained against a 4-bit base and you merge into 16-bit weights, the merged model is not the model you evaluated. Sometimes the difference is negligible and sometimes it is not.

Run your task evaluation on the merged artefact before it goes anywhere. If you then quantise the merged model to 4-bit for serving, which is the common path, run it again on the quantised version, because you have now applied two transformations to something you validated once.

The sequence that keeps you honest: evaluate the adapter as served, merge, evaluate, quantise, evaluate. Three runs of a small task set is an hour of compute and it is the difference between shipping a measured artefact and a hopeful one.

Version the whole thing, not the adapter

An adapter is meaningless on its own. Attached to the wrong base (even a different revision of the same model) it produces output that is subtly wrong and never errors, which is the worst failure shape there is.

Record, next to the adapter file, the base model repository and its exact commit hash, the tokeniser revision, the chat template used during training, the rank and alpha, the training data hash, and the evaluation score at that checkpoint. Treat those together as one artefact with one version number.

Text

adapters/triage/v3/
  adapter_model.safetensors
  adapter_config.json
  MANIFEST.md        # base repo + commit, template, data hash, eval score, date

Rolling it out

Because a merged model is an ordinary model, the deployment pattern from the serving module applies unchanged: bring the new one up on a second port, send it real traffic in shadow or to a small share of users, compare, then move the gateway’s model mapping. With separate adapters it is easier still: register the new adapter under a new name and change one line to route to it.

Keep the previous version loadable for at least a week. The problems a fine-tune introduces tend to surface on the inputs that occur a few times a day, not on the ones you tested.

Every lesson here is open. Your progress is recorded once you pass the Open Model Ecosystem exam.

A team evaluates a QLoRA adapter served on its 4-bit base, merges it into the 16-bit base, converts the result to a 4-bit GGUF, and deploys. Users report quality below what the evaluation showed. What was the process error?

Pick the one you would defend. Your answer stays private.