{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"serving-and-merging-adapters","title":"Serving what you made, merged or not","url":"https://addaly.com/learn/running-models-yourself/serving-and-merging-adapters","position":80,"of":82,"minutes":8,"answer":"Keeping adapters separate serves many tasks from one base in VRAM at a small per-token cost.","keyIdea":"Keeping adapters separate serves many tasks from one base in VRAM at a small per-token cost, while merging produces an ordinary model that any engine can load. And because a merge and a subsequent quantisation each change the artefact, evaluate after every transformation and version the adapter together with its exact base commit and template."},"body":{"format":"markdown","text":"## Two ways to deploy, with different properties\n\nAn adapter can stay separate or be folded into the weights. The choice is not about quality; it is about how many adapters you have and where they need to run.\n\n**Keep it separate.** The base model sits in VRAM once, and adapters are applied at runtime. vLLM supports this directly:\n\n```bash\nvllm serve Qwen/Qwen2.5-7B-Instruct \\\n  --enable-lora --max-loras 4 --max-lora-rank 32 \\\n  --lora-modules triage=/srv/adapters/triage support=/srv/adapters/support\n```\n\nCallers then ask for `triage` or `support` as if they were separate models, and both are served from one copy of the base. For a team with several task adapters this is a large memory saving: one 14GB base and three 40MB adapters instead of three 14GB models.\n\n**Merge it.** Fold the update into the weights and produce a single ordinary model:\n\n```python\nfrom peft import PeftModel\nbase = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)\nmerged = PeftModel.from_pretrained(base, \"out/checkpoint-150\").merge_and_unload()\nmerged.save_pretrained(\"merged-model\"); tok.save_pretrained(\"merged-model\")\n```\n\nThe arithmetic is exactly the formula from earlier: each adapted matrix becomes `W + (alpha/r) * B @ A`, computed once. The result is a normal model that any engine can load, that can be converted to GGUF and quantised, and that carries no per-token adapter overhead. This is the right choice when one adapter is the product, when it must run in llama.cpp or Ollama, or when it has to work on a machine with no adapter support at all.\n\n## The cost of keeping them separate\n\nMulti-adapter serving is not free. Each token's computation includes the low-rank product, which is a small overhead per request. More significantly, a batch containing requests for several different adapters is less efficient than a batch for one, because the adapter work cannot be shared across them. With a handful of adapters and modest traffic this is invisible; with dozens of adapters and heavy concurrency it is measurable.\n\nThere is also a hard constraint people meet late: `--max-lora-rank` must be at least the rank of every adapter you intend to load. An adapter trained at rank 64 will not load on a server configured for 16, and the error arrives when someone tries to use it rather than at startup.\n\n## Merging after QLoRA, again\n\nThe caution from the QLoRA lesson lands here. If the adapter was trained against a 4-bit base and you merge into 16-bit weights, the merged model is not the model you evaluated. Sometimes the difference is negligible and sometimes it is not.\n\nRun your task evaluation on the merged artefact before it goes anywhere. If you then quantise the merged model to 4-bit for serving, which is the common path, run it again on the quantised version, because you have now applied two transformations to something you validated once.\n\nThe sequence that keeps you honest: evaluate the adapter as served, merge, evaluate, quantise, evaluate. Three runs of a small task set is an hour of compute and it is the difference between shipping a measured artefact and a hopeful one.\n\n## Version the whole thing, not the adapter\n\nAn adapter is meaningless on its own. Attached to the wrong base (even a different revision of the same model) it produces output that is subtly wrong and never errors, which is the worst failure shape there is.\n\nRecord, next to the adapter file, the base model repository and its exact commit hash, the tokeniser revision, the chat template used during training, the rank and alpha, the training data hash, and the evaluation score at that checkpoint. Treat those together as one artefact with one version number.\n\n```\nadapters/triage/v3/\n  adapter_model.safetensors\n  adapter_config.json\n  MANIFEST.md        # base repo + commit, template, data hash, eval score, date\n```\n\n## Rolling it out\n\nBecause a merged model is an ordinary model, the deployment pattern from the serving module applies unchanged: bring the new one up on a second port, send it real traffic in shadow or to a small share of users, compare, then move the gateway's model mapping. With separate adapters it is easier still: register the new adapter under a new name and change one line to route to it.\n\nKeep the previous version loadable for at least a week. The problems a fine-tune introduces tend to surface on the inputs that occur a few times a day, not on the ones you tested."},"check":{"question":"A team evaluates a QLoRA adapter served on its 4-bit base, merges it into the 16-bit base, converts the result to a 4-bit GGUF, and deploys. Users report quality below what the evaluation showed. What was the process error?","options":[{"text":"Merging a QLoRA adapter is invalid and it should only ever be served against its quantised base.","correct":false,"tempting":"Overstates a real caution into a prohibition; merging is a standard and supported path provided the merged artefact is re-measured."},{"text":"The GGUF conversion loses adapter information, so the merge should have been performed after conversion instead.","correct":false,"tempting":"Conversion operates on merged weights and preserves them; the ordering suggested here is not possible with the formats involved."},{"text":"Two transformations were applied after the only evaluation, so the deployed model was never measured.","correct":true},{"text":"The adapter rank exceeded what the serving engine supported, so it was silently ignored at load time.","correct":false,"tempting":"A rank mismatch causes a load failure rather than silent omission, and it does not apply to a merged model, which contains no adapter."}]},"prev":{"slug":"reading-the-loss-curve","title":"Generalisation, memorisation and forgetting","url":"https://addaly.com/learn/running-models-yourself/reading-the-loss-curve"},"next":{"slug":"continued-pretraining","title":"When an adapter is not enough","url":"https://addaly.com/learn/running-models-yourself/continued-pretraining"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Serving what you made, merged or not', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/serving-and-merging-adapters (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Serving what you made, merged or not","courseTitle":"Running Models Yourself","module":{"index":9,"slug":"making-it-yours","title":"Making it yours: adapters and training"},"answer":null,"keyIdea":"Keeping adapters separate serves many tasks from one base in VRAM at a small per-token cost, while merging produces an ordinary model that any engine can load. And because a merge and a subsequent quantisation each change the artefact, evaluate after every transformation and version the adapter together with its exact base commit and template.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-ways-to-deploy-with-different-properties","runs":[{"text":"Two ways to deploy, with different properties"}]},{"t":"p","runs":[{"text":"An adapter can stay separate or be folded into the weights. The choice is not about quality; it is about how many adapters you have and where they need to run."}]},{"t":"p","runs":[{"text":"Keep it separate.","b":true},{"text":" The base model sits in VRAM once, and adapters are applied at runtime. vLLM supports this directly:"}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve Qwen/Qwen2.5-7B-Instruct \\\n  --enable-lora --max-loras 4 --max-lora-rank 32 \\\n  --lora-modules triage=/srv/adapters/triage support=/srv/adapters/support"},{"t":"p","runs":[{"text":"Callers then ask for "},{"text":"triage","code":true},{"text":" or "},{"text":"support","code":true},{"text":" as if they were separate models, and both are served from one copy of the base. For a team with several task adapters this is a large memory saving: one 14GB base and three 40MB adapters instead of three 14GB models."}]},{"t":"p","runs":[{"text":"Merge it.","b":true},{"text":" Fold the update into the weights and produce a single ordinary model:"}]},{"t":"code","lang":"python","label":"Python","text":"from peft import PeftModel\nbase = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)\nmerged = PeftModel.from_pretrained(base, \"out/checkpoint-150\").merge_and_unload()\nmerged.save_pretrained(\"merged-model\"); tok.save_pretrained(\"merged-model\")"},{"t":"p","runs":[{"text":"The arithmetic is exactly the formula from earlier: each adapted matrix becomes "},{"text":"W + (alpha/r) * B @ A","code":true},{"text":", computed once. The result is a normal model that any engine can load, that can be converted to GGUF and quantised, and that carries no per-token adapter overhead. This is the right choice when one adapter is the product, when it must run in llama.cpp or Ollama, or when it has to work on a machine with no adapter support at all."}]},{"t":"h2","id":"the-cost-of-keeping-them-separate","runs":[{"text":"The cost of keeping them separate"}]},{"t":"p","runs":[{"text":"Multi-adapter serving is not free. Each token’s computation includes the low-rank product, which is a small overhead per request. More significantly, a batch containing requests for several different adapters is less efficient than a batch for one, because the adapter work cannot be shared across them. With a handful of adapters and modest traffic this is invisible; with dozens of adapters and heavy concurrency it is measurable."}]},{"t":"p","runs":[{"text":"There is also a hard constraint people meet late: "},{"text":"--max-lora-rank","code":true},{"text":" must be at least the rank of every adapter you intend to load. An adapter trained at rank 64 will not load on a server configured for 16, and the error arrives when someone tries to use it rather than at startup."}]},{"t":"h2","id":"merging-after-qlora-again","runs":[{"text":"Merging after QLoRA, again"}]},{"t":"p","runs":[{"text":"The caution from the QLoRA lesson lands here. If the adapter was trained against a 4-bit base and you merge into 16-bit weights, the merged model is not the model you evaluated. Sometimes the difference is negligible and sometimes it is not."}]},{"t":"p","runs":[{"text":"Run your task evaluation on the merged artefact before it goes anywhere. If you then quantise the merged model to 4-bit for serving, which is the common path, run it again on the quantised version, because you have now applied two transformations to something you validated once."}]},{"t":"p","runs":[{"text":"The sequence that keeps you honest: evaluate the adapter as served, merge, evaluate, quantise, evaluate. Three runs of a small task set is an hour of compute and it is the difference between shipping a measured artefact and a hopeful one."}]},{"t":"h2","id":"version-the-whole-thing-not-the-adapter","runs":[{"text":"Version the whole thing, not the adapter"}]},{"t":"p","runs":[{"text":"An adapter is meaningless on its own. Attached to the wrong base (even a different revision of the same model) it produces output that is subtly wrong and never errors, which is the worst failure shape there is."}]},{"t":"p","runs":[{"text":"Record, next to the adapter file, the base model repository and its exact commit hash, the tokeniser revision, the chat template used during training, the rank and alpha, the training data hash, and the evaluation score at that checkpoint. Treat those together as one artefact with one version number."}]},{"t":"code","lang":"","label":"Text","text":"adapters/triage/v3/\n  adapter_model.safetensors\n  adapter_config.json\n  MANIFEST.md        # base repo + commit, template, data hash, eval score, date"},{"t":"h2","id":"rolling-it-out","runs":[{"text":"Rolling it out"}]},{"t":"p","runs":[{"text":"Because a merged model is an ordinary model, the deployment pattern from the serving module applies unchanged: bring the new one up on a second port, send it real traffic in shadow or to a small share of users, compare, then move the gateway’s model mapping. With separate adapters it is easier still: register the new adapter under a new name and change one line to route to it."}]},{"t":"p","runs":[{"text":"Keep the previous version loadable for at least a week. The problems a fine-tune introduces tend to surface on the inputs that occur a few times a day, not on the ones you tested."}]}],"check":{"question":[{"text":"A team evaluates a QLoRA adapter served on its 4-bit base, merges it into the 16-bit base, converts the result to a 4-bit GGUF, and deploys. Users report quality below what the evaluation showed. What was the process error?"}],"options":[{"text":[{"text":"Merging a QLoRA adapter is invalid and it should only ever be served against its quantised base."}],"correct":false,"tempting":[{"text":"Overstates a real caution into a prohibition; merging is a standard and supported path provided the merged artefact is re-measured."}]},{"text":[{"text":"The GGUF conversion loses adapter information, so the merge should have been performed after conversion instead."}],"correct":false,"tempting":[{"text":"Conversion operates on merged weights and preserves them; the ordering suggested here is not possible with the formats involved."}]},{"text":[{"text":"Two transformations were applied after the only evaluation, so the deployed model was never measured."}],"correct":true,"tempting":null},{"text":[{"text":"The adapter rank exceeded what the serving engine supported, so it was silently ignored at load time."}],"correct":false,"tempting":[{"text":"A rank mismatch causes a load failure rather than silent omission, and it does not apply to a merged model, which contains no adapter."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"reading-the-loss-curve","title":"Generalisation, memorisation and forgetting","minutes":9},"next":{"slug":"continued-pretraining","title":"When an adapter is not enough","minutes":9}}}