Serving the Adapter

What you actually have

After a LoRA run the artifact is small:

out/checkpoint-300/
  adapter_model.safetensors   # 40-200 MB
  adapter_config.json         # rank, alpha, target modules, base model id

It is meaningless without the exact base model. adapter_config.json records the base repository name but not necessarily the revision, so pin the commit hash yourself in your own config. Base repositories do get updated in place. An adapter trained against last month's weights and served against this month's is a subtle quality bug that nobody will diagnose.

Merge, or keep it separate

Serve the adapter separately when you want to A/B against the base, ship several variants, or turn the fine-tune off without a redeploy. vLLM does this natively:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-lora \
  --lora-modules support=/models/support-lora \
  --max-lora-rank 16

Then request "model": "support", or the base name for the base. One GPU, one copy of the base weights, many adapters: this is how you serve a per-customer model without a GPU per customer. Expect a modest throughput cost, in the region of 10-20%, growing with how many adapters are live at once.

Merge when you have one variant and want no overhead:

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16")
merged = PeftModel.from_pretrained(base, "out/checkpoint-300").merge_and_unload()
merged.save_pretrained("merged-7b")

You get an ordinary model any server can load, at the cost of a full 14 GB artifact per variant and no ability to stack adapters.

One caveat that bites: if you trained with QLoRA, the adapter learned as a correction to a 4-bit base. Merging it into the 16-bit base is not the same arithmetic. It usually works, and it sometimes moves quality by a point or two — a specific case of the general rule below.

Evaluate the artifact you ship

This is the most-skipped step in the whole pipeline.

Every transformation after training changes the model: merging, quantizing to GGUF or AWQ or GPTQ, a different inference engine, different sampling defaults, a chat template the server applies for you. Any of them can move your numbers, and none of them announce it.

So run your held-out evaluation *through the endpoint you will actually serve*, with production sampling settings, and compare it to the number from the training rig. Same inputs, same judge. If it dropped, you found out before your users did.

Running it locally

For CPU or laptop deployment: merge, convert, quantize.

python convert_hf_to_gguf.py ./merged-7b --outfile model.f16.gguf
./llama-quantize model.f16.gguf model.Q4_K_M.gguf Q4_K_M

Q4_K_M on a 7B gives roughly a 4 GB file that runs in 8 GB of RAM at a few tokens per second on a modern CPU. That is usable for batch work and personal tools, and slow for interactive chat. Then a Modelfile for Ollama:

FROM ./model.Q4_K_M.gguf
TEMPLATE """..."""

Set the template explicitly to the one you trained with. Do not accept a default.

The cost question, honestly

A 24 GB GPU running continuously costs roughly $300-700 a month depending on provider and commitment. That buys a great many tokens from a hosted API. Self-serving a fine-tuned model wins when you have steady volume to keep the GPU busy, when the data has to stay on your infrastructure, when you need predictable latency, or when the behaviour genuinely cannot be prompted.

If your traffic is spiky and modest, a hosted endpoint for your adapter — or no fine-tune at all — is the cheaper engineering decision, and saying so is not a failure.

Before you ship

• Base model repository and commit hash pinned.
• Chat template pinned in the serving config, matching training.
• Held-out evaluation re-run against the real endpoint.
• Canary set re-run against the real endpoint.
• The base model still deployed, one config flag away.