Merging models, and what it really does

Averaging weights, and why it is not absurd

You can take two fine-tunes of the same base model and average their weights. The result often works, and sometimes performs better than either parent.

That sounds like it should produce nonsense, and the reason it does not is worth knowing: fine-tunes of a shared base stay in a connected low-loss region of the weight space. They have not travelled far from the base or from each other, so points between them are also reasonable models. Merge two models with different bases and you do get nonsense, immediately.

The hard requirements: identical architecture, identical tokenizer, same shapes. In practice, a shared base model.

The methods

Linear averaging and SLERP interpolate directly between two sets of weights, SLERP along a spherical path rather than a straight line. Simple and effective for two models.

Task arithmetic is the idea that clarifies the rest. Define a task vector as the difference between a fine-tune and its base: the change tuning made. Task vectors can be added to combine capabilities, or subtracted to remove one. This gives a vocabulary for what merging is: you are doing arithmetic on the deltas, not on the models.

TIES merges many models by trimming small changes, resolving sign conflicts by a weighted vote, and averaging only the parameters that agree. It handles interference between task vectors, which plain averaging does not.

DARE randomly drops most of the delta and rescales what remains, on the observation that fine-tuning deltas are highly redundant. Often combined with TIES.

mergekit implements all of these from a short configuration:

models:
  - model: ./my-support-tune
    parameters: {weight: 0.5, density: 0.6}
  - model: ./my-multilingual-tune
    parameters: {weight: 0.5, density: 0.6}
merge_method: ties
base_model: Qwen/Qwen3-8B
dtype: bfloat16

Merging is cheap. It runs on a CPU with enough RAM, takes minutes, and needs no training data.

The use that reliably works

Merging your own runs. Train the same configuration with three seeds, or take several checkpoints from one run, and average them. This is the model-soup idea and it is the most dependable form of merging: it reduces variance, usually gains a little, and costs nothing but disk.

Combining your own domain adapters, where each was trained on a distinct capability you want together. Test the combination on both tasks; interference is common and TIES exists because of it.

The use that should make you cautious

Public leaderboards have been full of merges of merges, tuned by trying combinations until the score went up. That process optimises the benchmark directly, and the resulting models frequently underperform their headline numbers on anything else.

There are three further problems with a deep merge chain, and they are the reasons the vetting lesson treated them as a red flag:

• Licence. The merged model carries the obligations of every parent. A single non-commercial ancestor makes the whole thing non-commercial.
• Provenance. Nobody can say what data influenced the result, so no claim about the model’s behaviour is supportable.
• Evaluation. Merges are frequently selected on the benchmark they are then reported against, which is the definition of overfitting.

How to merge responsibly

1. Merge models whose parents you know and whose licences you have checked.
2. Evaluate on a held-out set the merge selection did not use.
3. Include the regression set from the previous lesson; merges shift refusal behaviour and language coverage in ways nobody intends.
4. Record the recipe (the exact configuration, the parent revisions, the method) so the result is reproducible.
5. Prefer merging your own runs over merging strangers’ models.

Merging is a useful tool with an unusually low cost and an unusually high rate of being used badly. The discipline that separates the two is entirely in step 2.