{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"serving-the-adapter","title":"Serving the Adapter","url":"https://addaly.com/learn/fine-tuning/serving-the-adapter","position":76,"of":84,"minutes":9,"answer":"Merging, quantizing and switching engines all change the model; evaluate the exact artifact you serve.","keyIdea":"Merging, quantizing and switching engines all change the model; evaluate the exact artifact you serve."},"body":{"format":"markdown","text":"## What you actually have\n\nAfter a LoRA run the artifact is small:\n\n```\nout/checkpoint-300/\n  adapter_model.safetensors   # 40-200 MB\n  adapter_config.json         # rank, alpha, target modules, base model id\n```\n\nIt is meaningless without the exact base model. `adapter_config.json` records the base repository name but not necessarily the revision, so pin the commit hash yourself in your own config. Base repositories do get updated in place. An adapter trained against last month's weights and served against this month's is a subtle quality bug that nobody will diagnose.\n\n## Merge, or keep it separate\n\n**Serve the adapter separately** when you want to A/B against the base, ship several variants, or turn the fine-tune off without a redeploy. vLLM does this natively:\n\n```bash\nvllm serve Qwen/Qwen2.5-7B-Instruct \\\n  --enable-lora \\\n  --lora-modules support=/models/support-lora \\\n  --max-lora-rank 16\n```\n\nThen request `\"model\": \"support\"`, or the base name for the base. One GPU, one copy of the base weights, many adapters: this is how you serve a per-customer model without a GPU per customer. Expect a throughput cost that grows with rank and with how many adapters are live at once; published measurements run from small to about half, so measure your own.\n\n**Merge** when you have one variant and want no overhead:\n\n```python\nfrom peft import PeftModel\nfrom transformers import AutoModelForCausalLM\nbase = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=\"bfloat16\")\nmerged = PeftModel.from_pretrained(base, \"out/checkpoint-300\").merge_and_unload()\nmerged.save_pretrained(\"merged-7b\")\n```\n\nYou get an ordinary model any server can load, at the cost of a full 14 GB artifact per variant and no ability to stack adapters.\n\nOne caveat that bites: if you trained with QLoRA, the adapter learned as a correction to a **4-bit** base. Merging it into the 16-bit base is not the same arithmetic. It usually works, and it sometimes moves quality by a point or two: a specific case of the general rule below.\n\n## Evaluate the artifact you ship\n\nThis is the most-skipped step in the whole pipeline.\n\nEvery transformation after training changes the model: merging, quantizing to GGUF or AWQ or GPTQ, a different inference engine, different sampling defaults, a chat template the server applies for you. Any of them can move your numbers, and none of them announce it.\n\nSo run your held-out evaluation *through the endpoint you will actually serve*, with production sampling settings, and compare it to the number from the training rig. Same inputs, same judge. If it dropped, you found out before your users did.\n\n## Running it locally\n\nFor CPU or laptop deployment: merge, convert, quantize.\n\n```bash\npython convert_hf_to_gguf.py ./merged-7b --outfile model.f16.gguf\n./llama-quantize model.f16.gguf model.Q4_K_M.gguf Q4_K_M\n```\n\nQ4_K_M on a 7B gives roughly a 4 GB file that runs in 8 GB of RAM at a few tokens per second on a modern CPU. That is usable for batch work and personal tools, and slow for interactive chat. Then a Modelfile for Ollama:\n\n> Figure: Merge it in, or keep it separate. Keep the adapter separate: one base in memory, many adapters; turn the fine-tune off with a flag, not a redeploy; A/B two versions against the same base; costs some throughput: measure it on your own traffic. Merge it into the base: an ordinary model any server loads; no adapter overhead at all; a full 14 GB artefact per variant, and no stacking; a QLoRA adapter was fitted against a 4-bit base, so merging into the 16-bit one is not the same arithmetic. Whichever you choose, run the held-out and canary sets through the endpoint you will actually serve, with production sampling settings. Every transformation after training changes the model and none of them announce it.\n\n```\nFROM ./model.Q4_K_M.gguf\nTEMPLATE \"\"\"...\"\"\"\n```\n\nSet the template explicitly to the one you trained with. Do not accept a default.\n\n## The cost question\n\nA 24 GB GPU running continuously costs roughly $300-700 a month depending on provider and commitment. That buys a great many tokens from a hosted API. Self-serving a fine-tuned model wins when you have steady volume to keep the GPU busy, when the data has to stay on your infrastructure, when you need predictable latency, or when the behaviour cannot be prompted.\n\nIf your traffic is spiky and modest, a hosted endpoint for your adapter, or no fine-tune at all, is the cheaper engineering decision, and saying so is not a failure.\n\n## Before you ship\n\n- Base model repository and commit hash pinned.\n- Chat template pinned in the serving config, matching training.\n- Held-out evaluation re-run against the real endpoint.\n- Canary set re-run against the real endpoint.\n- The base model still deployed, one config flag away."},"check":{"question":"You merged a QLoRA adapter into the 16-bit base, converted to 4-bit GGUF, and deployed on Ollama. The house style is clearly present, but accuracy is below the number you recorded after training. What do you do first?","options":[{"text":"Requantize at Q8_0, since 4-bit quantization is the most likely culprit.","correct":false,"tempting":"Quantization is a reasonable suspect and Q8 may well help, but you would be swapping parts without a measurement and still would not know what moved."},{"text":"Retrain at a higher rank; the adapter was probably underpowered.","correct":false,"tempting":"Rank is a real quality lever, but nothing here implicates training: the tuned style survived, so the adapter is present and working."},{"text":"Go back to serving the unmerged adapter with vLLM and treat the GGUF path as broken.","correct":false,"tempting":"Reverting to a known-good configuration is sound instinct during an incident, but it forfeits the CPU deployment you needed and leaves the cause unmeasured."},{"text":"Re-run the held-out evaluation through the Ollama endpoint with production sampling settings, so you are measuring the artifact users actually get.","correct":true}]},"prev":{"slug":"ablations-that-earn-their-cost","title":"Ablations that earn their cost","url":"https://addaly.com/learn/fine-tuning/ablations-that-earn-their-cost"},"next":{"slug":"quantising-for-serving","title":"Quantising the model you ship","url":"https://addaly.com/learn/fine-tuning/quantising-for-serving"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Serving the Adapter', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/serving-the-adapter (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Serving the Adapter","courseTitle":"Fine-Tuning, and When Not To","module":{"index":9,"slug":"shipping-and-living-with-it","title":"Shipping it, and living with it"},"answer":null,"keyIdea":"Merging, quantizing and switching engines all change the model; evaluate the exact artifact you serve.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 9.1","figure":{"left":{"title":"Keep the adapter separate","items":["one base in memory, many adapters","turn the fine-tune off with a flag, not a redeploy","A/B two versions against the same base","costs some throughput: measure it on your own traffic"]},"type":"compare","right":{"title":"Merge it into the base","items":["an ordinary model any server loads","no adapter overhead at all","a full 14 GB artefact per variant, and no stacking","a QLoRA adapter was fitted against a 4-bit base, so merging into the 16-bit one is not the same arithmetic"]},"title":"Merge it in, or keep it separate","caption":"Whichever you choose, run the held-out and canary sets through the endpoint you will actually serve, with production sampling settings. Every transformation after training changes the model and none of them announce it."},"alt":"Figure: Merge it in, or keep it separate. Keep the adapter separate: one base in memory, many adapters; turn the fine-tune off with a flag, not a redeploy; A/B two versions against the same base; costs some throughput: measure it on your own traffic. Merge it into the base: an ordinary model any server loads; no adapter overhead at all; a full 14 GB artefact per variant, and no stacking; a QLoRA adapter was fitted against a 4-bit base, so merging into the 16-bit one is not the same arithmetic. Whichever you choose, run the held-out and canary sets through the endpoint you will actually serve, with production sampling settings. Every transformation after training changes the model and none of them announce it."}},"blocks":[{"t":"h2","id":"what-you-actually-have","runs":[{"text":"What you actually have"}]},{"t":"p","runs":[{"text":"After a LoRA run the artifact is small:"}]},{"t":"code","lang":"","label":"Text","text":"out/checkpoint-300/\n  adapter_model.safetensors   # 40-200 MB\n  adapter_config.json         # rank, alpha, target modules, base model id"},{"t":"p","runs":[{"text":"It is meaningless without the exact base model. "},{"text":"adapter_config.json","code":true},{"text":" records the base repository name but not necessarily the revision, so pin the commit hash yourself in your own config. Base repositories do get updated in place. An adapter trained against last month’s weights and served against this month’s is a subtle quality bug that nobody will diagnose."}]},{"t":"h2","id":"merge-or-keep-it-separate","runs":[{"text":"Merge, or keep it separate"}]},{"t":"p","runs":[{"text":"Serve the adapter separately","b":true},{"text":" when you want to A/B against the base, ship several variants, or turn the fine-tune off without a redeploy. vLLM does this natively:"}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve Qwen/Qwen2.5-7B-Instruct \\\n  --enable-lora \\\n  --lora-modules support=/models/support-lora \\\n  --max-lora-rank 16"},{"t":"p","runs":[{"text":"Then request "},{"text":"\"model\": \"support\"","code":true},{"text":", or the base name for the base. One GPU, one copy of the base weights, many adapters: this is how you serve a per-customer model without a GPU per customer. Expect a throughput cost that grows with rank and with how many adapters are live at once; published measurements run from small to about half, so measure your own."}]},{"t":"p","runs":[{"text":"Merge","b":true},{"text":" when you have one variant and want no overhead:"}]},{"t":"code","lang":"python","label":"Python","text":"from peft import PeftModel\nfrom transformers import AutoModelForCausalLM\nbase = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=\"bfloat16\")\nmerged = PeftModel.from_pretrained(base, \"out/checkpoint-300\").merge_and_unload()\nmerged.save_pretrained(\"merged-7b\")"},{"t":"p","runs":[{"text":"You get an ordinary model any server can load, at the cost of a full 14 GB artifact per variant and no ability to stack adapters."}]},{"t":"p","runs":[{"text":"One caveat that bites: if you trained with QLoRA, the adapter learned as a correction to a "},{"text":"4-bit","b":true},{"text":" base. Merging it into the 16-bit base is not the same arithmetic. It usually works, and it sometimes moves quality by a point or two: a specific case of the general rule below."}]},{"t":"h2","id":"evaluate-the-artifact-you-ship","runs":[{"text":"Evaluate the artifact you ship"}]},{"t":"p","runs":[{"text":"This is the most-skipped step in the whole pipeline."}]},{"t":"p","runs":[{"text":"Every transformation after training changes the model: merging, quantizing to GGUF or AWQ or GPTQ, a different inference engine, different sampling defaults, a chat template the server applies for you. Any of them can move your numbers, and none of them announce it."}]},{"t":"p","runs":[{"text":"So run your held-out evaluation "},{"text":"through the endpoint you will actually serve","i":true},{"text":", with production sampling settings, and compare it to the number from the training rig. Same inputs, same judge. If it dropped, you found out before your users did."}]},{"t":"h2","id":"running-it-locally","runs":[{"text":"Running it locally"}]},{"t":"p","runs":[{"text":"For CPU or laptop deployment: merge, convert, quantize."}]},{"t":"code","lang":"bash","label":"Bash","text":"python convert_hf_to_gguf.py ./merged-7b --outfile model.f16.gguf\n./llama-quantize model.f16.gguf model.Q4_K_M.gguf Q4_K_M"},{"t":"p","runs":[{"text":"Q4_K_M on a 7B gives roughly a 4 GB file that runs in 8 GB of RAM at a few tokens per second on a modern CPU. That is usable for batch work and personal tools, and slow for interactive chat. Then a Modelfile for Ollama:"}]},{"t":"code","lang":"","label":"Text","text":"FROM ./model.Q4_K_M.gguf\nTEMPLATE \"\"\"...\"\"\""},{"t":"p","runs":[{"text":"Set the template explicitly to the one you trained with. Do not accept a default."}]},{"t":"h2","id":"the-cost-question","runs":[{"text":"The cost question"}]},{"t":"p","runs":[{"text":"A 24 GB GPU running continuously costs roughly $300-700 a month depending on provider and commitment. That buys a great many tokens from a hosted API. Self-serving a fine-tuned model wins when you have steady volume to keep the GPU busy, when the data has to stay on your infrastructure, when you need predictable latency, or when the behaviour cannot be prompted."}]},{"t":"p","runs":[{"text":"If your traffic is spiky and modest, a hosted endpoint for your adapter, or no fine-tune at all, is the cheaper engineering decision, and saying so is not a failure."}]},{"t":"h2","id":"before-you-ship","runs":[{"text":"Before you ship"}]},{"t":"ul","items":[[{"text":"Base model repository and commit hash pinned."}],[{"text":"Chat template pinned in the serving config, matching training."}],[{"text":"Held-out evaluation re-run against the real endpoint."}],[{"text":"Canary set re-run against the real endpoint."}],[{"text":"The base model still deployed, one config flag away."}]]}],"check":{"question":[{"text":"You merged a QLoRA adapter into the 16-bit base, converted to 4-bit GGUF, and deployed on Ollama. The house style is clearly present, but accuracy is below the number you recorded after training. What do you do first?"}],"options":[{"text":[{"text":"Requantize at Q8_0, since 4-bit quantization is the most likely culprit."}],"correct":false,"tempting":[{"text":"Quantization is a reasonable suspect and Q8 may well help, but you would be swapping parts without a measurement and still would not know what moved."}]},{"text":[{"text":"Retrain at a higher rank; the adapter was probably underpowered."}],"correct":false,"tempting":[{"text":"Rank is a real quality lever, but nothing here implicates training: the tuned style survived, so the adapter is present and working."}]},{"text":[{"text":"Go back to serving the unmerged adapter with vLLM and treat the GGUF path as broken."}],"correct":false,"tempting":[{"text":"Reverting to a known-good configuration is sound instinct during an incident, but it forfeits the CPU deployment you needed and leaves the cause unmeasured."}]},{"text":[{"text":"Re-run the held-out evaluation through the Ollama endpoint with production sampling settings, so you are measuring the artifact users actually get."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"ablations-that-earn-their-cost","title":"Ablations that earn their cost","minutes":8},"next":{"slug":"quantising-for-serving","title":"Quantising the model you ship","minutes":9}}}