Mixtures and replay

Why a pure task dataset is a trap

Train a model on 4,000 examples that are all ticket triage, and you get a model that does ticket triage. You also get a model that has drifted towards producing triage-shaped output when asked anything, that has become worse at ordinary instruction-following, and that may have lost the refusal behaviour its base model had.

This is catastrophic forgetting, and the most effective countermeasure is not a training trick. It is putting other things in the data.

Replay, and the ratio

Replay means mixing examples of the general capability you want preserved into your task-specific set. A working starting point:

• 70 to 85% task data
• 15 to 30% general instruction data

The general portion comes from a permissively licensed public set (OpenAssistant’s Apache-2.0 conversations are the usual choice), or, better where you can get it, from logs of the model doing the other things your product asks of it.

The mechanism is direct. Gradient descent moves weights towards whatever reduces loss on the batch. If every batch contains only triage, every weight is free to specialise. If a fifth of each batch is ordinary conversation, the weights that support ordinary conversation keep receiving a signal to stay where they are.

This costs nothing but dataset assembly, and in measured comparisons it does more to preserve general capability than lowering the learning rate, reducing epochs, or any adapter configuration.

Getting the ratio right

Run the ablation; it is two extra runs:

100% task        → task metric 0.91, canary 0.62   (was 0.84 before tuning)
80/20 mixture    → task metric 0.90, canary 0.81
60/40 mixture    → task metric 0.86, canary 0.83

That shape (a large canary recovery for almost no task cost at 80/20, then diminishing returns) is what you usually see. The 80/20 row is the one to ship, and you only know it is there because you ran three configurations instead of one.

Balancing within the task data

The mixture question also applies inside your own data. If 70% of your traffic is one intent, and you train on traffic proportions, the model will be excellent at that intent and mediocre at the rest, which is a defensible choice, but it should be a choice.

Three options:

1. Natural proportions. The model’s competence matches your traffic. Simple, and it optimises expected quality.
2. Flattened. Cap any single intent at some share (say 25%) so rare intents get enough examples to be learned. Costs a little on the head, buys a lot on the tail.
3. Weighted by cost of failure. If getting a refund wrong costs more than getting a shipping query wrong, over-represent refunds regardless of volume.

Most teams want option 2 and implement option 1 by not thinking about it.

Curriculum: mostly not worth it

Ordering examples from easy to hard has a long research history and produces inconsistent results on LLM fine-tuning. What is not optional is shuffling: if your file is sorted by source, or by date, or by intent, the model sees one distribution for the first thousand steps and a different one for the next thousand. With a decaying learning rate, whatever comes last gets learned with the smallest steps and whatever comes first gets partially overwritten.

dataset = dataset.shuffle(seed=0)   # one line; not optional

This is a real bug that produces a mysteriously bad model, and the loss curve looks fine throughout.

Multi-task fine-tuning

If you have three tasks, one model trained on all three usually beats three separate adapters, provided the tasks are related. Shared structure transfers; the model learns your domain once rather than three times, and you maintain one artefact.

It stops being true when the tasks conflict: one wants terse structured output and another wants discursive explanation, and the model is asked to learn that the same kind of input maps to both. If your tasks pull in opposite directions, either separate them into different adapters or make the distinction explicit in the input: a system prompt or a tag that tells the model which mode it is in. Making it explicit is usually the better answer, because it keeps one artefact and gives you a controllable switch.

The rule to keep

Before you train, write down the proportions of your file by source and by intent, as actual counts. Almost nobody does this, and almost every surprising fine-tune result is explained by it.