{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"data-mixtures-and-replay","title":"Mixtures and replay","url":"https://addaly.com/learn/fine-tuning/data-mixtures-and-replay","position":34,"of":84,"minutes":9,"answer":"Mixing 15 to 30% general instruction data into a task-specific set preserves general capability better than any training-side mitigation, because the weights supporting that capability keep receiving a signal to stay put.","keyIdea":"Mixing 15 to 30% general instruction data into a task-specific set preserves general capability better than any training-side mitigation, because the weights supporting that capability keep receiving a signal to stay put; and shuffling the file is not optional when the learning rate decays."},"body":{"format":"markdown","text":"## Why a pure task dataset is a trap\n\nTrain a model on 4,000 examples that are all ticket triage, and you get a model that does ticket triage. You also get a model that has drifted towards producing triage-shaped output when asked anything, that has become worse at ordinary instruction-following, and that may have lost the refusal behaviour its base model had.\n\nThis is catastrophic forgetting, and the most effective countermeasure is not a training trick. It is putting other things in the data.\n\n## Replay, and the ratio\n\nReplay means mixing examples of the general capability you want preserved into your task-specific set. A working starting point:\n\n- **70 to 85% task data**\n- **15 to 30% general instruction data**\n\nThe general portion comes from a permissively licensed public set (OpenAssistant's Apache-2.0 conversations are the usual choice), or, better where you can get it, from logs of the model doing the *other* things your product asks of it.\n\nThe mechanism is direct. Gradient descent moves weights towards whatever reduces loss on the batch. If every batch contains only triage, every weight is free to specialise. If a fifth of each batch is ordinary conversation, the weights that support ordinary conversation keep receiving a signal to stay where they are.\n\nThis costs nothing but dataset assembly, and in measured comparisons it does more to preserve general capability than lowering the learning rate, reducing epochs, or any adapter configuration.\n\n## Getting the ratio right\n\nRun the ablation; it is two extra runs:\n\n```\n100% task        → task metric 0.91, canary 0.62   (was 0.84 before tuning)\n80/20 mixture    → task metric 0.90, canary 0.81\n60/40 mixture    → task metric 0.86, canary 0.83\n```\n\nThat shape (a large canary recovery for almost no task cost at 80/20, then diminishing returns) is what you usually see. The 80/20 row is the one to ship, and you only know it is there because you ran three configurations instead of one.\n\n## Balancing within the task data\n\nThe mixture question also applies inside your own data. If 70% of your traffic is one intent, and you train on traffic proportions, the model will be excellent at that intent and mediocre at the rest, which is a defensible choice, but it should be a choice.\n\nThree options:\n\n1. **Natural proportions.** The model's competence matches your traffic. Simple, and it optimises expected quality.\n2. **Flattened.** Cap any single intent at some share (say 25%) so rare intents get enough examples to be learned. Costs a little on the head, buys a lot on the tail.\n3. **Weighted by cost of failure.** If getting a refund wrong costs more than getting a shipping query wrong, over-represent refunds regardless of volume.\n\nMost teams want option 2 and implement option 1 by not thinking about it.\n\n## Curriculum: mostly not worth it\n\nOrdering examples from easy to hard has a long research history and produces inconsistent results on LLM fine-tuning. What is *not* optional is shuffling: if your file is sorted by source, or by date, or by intent, the model sees one distribution for the first thousand steps and a different one for the next thousand. With a decaying learning rate, whatever comes last gets learned with the smallest steps and whatever comes first gets partially overwritten.\n\n> Figure: The replay ablation, in three runs. Line chart of score against share of general instruction data in the mixture, per cent: Task metric (0, 0.91) (20, 0.9) (40, 0.86); Canary, general capability (0, 0.62) (20, 0.81) (40, 0.83). The canary was 0.84 before any tuning. A large recovery for almost no task cost at eighty-twenty, then diminishing returns, and you only know the row is there because you ran three configurations instead of one.\n\n```python\ndataset = dataset.shuffle(seed=0)   # one line; not optional\n```\n\nThis is a real bug that produces a mysteriously bad model, and the loss curve looks fine throughout.\n\n## Multi-task fine-tuning\n\nIf you have three tasks, one model trained on all three usually beats three separate adapters, provided the tasks are related. Shared structure transfers; the model learns your domain once rather than three times, and you maintain one artefact.\n\nIt stops being true when the tasks conflict: one wants terse structured output and another wants discursive explanation, and the model is asked to learn that the same kind of input maps to both. If your tasks pull in opposite directions, either separate them into different adapters or make the distinction explicit in the input: a system prompt or a tag that tells the model which mode it is in. Making it explicit is usually the better answer, because it keeps one artefact and gives you a controllable switch.\n\n## The rule to keep\n\nBefore you train, write down the proportions of your file by source and by intent, as actual counts. Almost nobody does this, and almost every surprising fine-tune result is explained by it."},"check":{"question":"A team's training file is ordered by source: 3,000 triage examples followed by 1,000 general conversations. They shuffle nothing and use a cosine schedule. Why is this worse than the same data shuffled?","options":[{"text":"Unshuffled data causes the optimiser's momentum buffers to saturate, which prevents later examples from producing gradients.","correct":false,"tempting":"Invents a saturation mechanism; momentum is an exponential average that adapts continuously and does not stop responding to new gradients."},{"text":"The general examples arrive last, where the learning rate is smallest, so they influence the weights least.","correct":true},{"text":"Batches drawn from a single source have lower gradient variance, so the effective learning rate is higher and the run diverges.","correct":false,"tempting":"Correctly notes within-batch homogeneity but draws the wrong conclusion; low variance does not raise the learning rate or cause divergence."},{"text":"The model will overfit the 3,000 triage examples in the first epoch before ever seeing general data, and later data cannot undo memorisation.","correct":false,"tempting":"Assumes memorisation is irreversible, when the real asymmetry is that the schedule gives the later data much smaller steps."}]},"prev":{"slug":"distillation-datasets-in-practice","title":"Generating a dataset from a teacher","url":"https://addaly.com/learn/fine-tuning/distillation-datasets-in-practice"},"next":{"slug":"packing-and-long-context","title":"Packing, document boundaries and long context","url":"https://addaly.com/learn/fine-tuning/packing-and-long-context"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Mixtures and replay', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/data-mixtures-and-replay (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Mixtures and replay","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Mixing 15 to 30% general instruction data into a task-specific set preserves general capability better than any training-side mitigation, because the weights supporting that capability keep receiving a signal to stay put; and shuffling the file is not optional when the learning rate decays.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The replay ablation, in three runs","series":[{"label":"Task metric","points":[[0,0.91],[20,0.9],[40,0.86]]},{"label":"Canary, general capability","points":[[0,0.62],[20,0.81],[40,0.83]]}],"xLabel":"share of general instruction data in the mixture, per cent","yLabel":"score","caption":"The canary was 0.84 before any tuning. A large recovery for almost no task cost at eighty-twenty, then diminishing returns, and you only know the row is there because you ran three configurations instead of one."},"alt":"Figure: The replay ablation, in three runs. Line chart of score against share of general instruction data in the mixture, per cent: Task metric (0, 0.91) (20, 0.9) (40, 0.86); Canary, general capability (0, 0.62) (20, 0.81) (40, 0.83). The canary was 0.84 before any tuning. A large recovery for almost no task cost at eighty-twenty, then diminishing returns, and you only know the row is there because you ran three configurations instead of one."}},"blocks":[{"t":"h2","id":"why-a-pure-task-dataset-is-a-trap","runs":[{"text":"Why a pure task dataset is a trap"}]},{"t":"p","runs":[{"text":"Train a model on 4,000 examples that are all ticket triage, and you get a model that does ticket triage. You also get a model that has drifted towards producing triage-shaped output when asked anything, that has become worse at ordinary instruction-following, and that may have lost the refusal behaviour its base model had."}]},{"t":"p","runs":[{"text":"This is catastrophic forgetting, and the most effective countermeasure is not a training trick. It is putting other things in the data."}]},{"t":"h2","id":"replay-and-the-ratio","runs":[{"text":"Replay, and the ratio"}]},{"t":"p","runs":[{"text":"Replay means mixing examples of the general capability you want preserved into your task-specific set. A working starting point:"}]},{"t":"ul","items":[[{"text":"70 to 85% task data","b":true}],[{"text":"15 to 30% general instruction data","b":true}]]},{"t":"p","runs":[{"text":"The general portion comes from a permissively licensed public set (OpenAssistant’s Apache-2.0 conversations are the usual choice), or, better where you can get it, from logs of the model doing the "},{"text":"other","i":true},{"text":" things your product asks of it."}]},{"t":"p","runs":[{"text":"The mechanism is direct. Gradient descent moves weights towards whatever reduces loss on the batch. If every batch contains only triage, every weight is free to specialise. If a fifth of each batch is ordinary conversation, the weights that support ordinary conversation keep receiving a signal to stay where they are."}]},{"t":"p","runs":[{"text":"This costs nothing but dataset assembly, and in measured comparisons it does more to preserve general capability than lowering the learning rate, reducing epochs, or any adapter configuration."}]},{"t":"h2","id":"getting-the-ratio-right","runs":[{"text":"Getting the ratio right"}]},{"t":"p","runs":[{"text":"Run the ablation; it is two extra runs:"}]},{"t":"code","lang":"","label":"Text","text":"100% task        → task metric 0.91, canary 0.62   (was 0.84 before tuning)\n80/20 mixture    → task metric 0.90, canary 0.81\n60/40 mixture    → task metric 0.86, canary 0.83"},{"t":"p","runs":[{"text":"That shape (a large canary recovery for almost no task cost at 80/20, then diminishing returns) is what you usually see. The 80/20 row is the one to ship, and you only know it is there because you ran three configurations instead of one."}]},{"t":"h2","id":"balancing-within-the-task-data","runs":[{"text":"Balancing within the task data"}]},{"t":"p","runs":[{"text":"The mixture question also applies inside your own data. If 70% of your traffic is one intent, and you train on traffic proportions, the model will be excellent at that intent and mediocre at the rest, which is a defensible choice, but it should be a choice."}]},{"t":"p","runs":[{"text":"Three options:"}]},{"t":"ol","start":1,"items":[[{"text":"Natural proportions.","b":true},{"text":" The model’s competence matches your traffic. Simple, and it optimises expected quality."}],[{"text":"Flattened.","b":true},{"text":" Cap any single intent at some share (say 25%) so rare intents get enough examples to be learned. Costs a little on the head, buys a lot on the tail."}],[{"text":"Weighted by cost of failure.","b":true},{"text":" If getting a refund wrong costs more than getting a shipping query wrong, over-represent refunds regardless of volume."}]]},{"t":"p","runs":[{"text":"Most teams want option 2 and implement option 1 by not thinking about it."}]},{"t":"h2","id":"curriculum-mostly-not-worth-it","runs":[{"text":"Curriculum: mostly not worth it"}]},{"t":"p","runs":[{"text":"Ordering examples from easy to hard has a long research history and produces inconsistent results on LLM fine-tuning. What is "},{"text":"not","i":true},{"text":" optional is shuffling: if your file is sorted by source, or by date, or by intent, the model sees one distribution for the first thousand steps and a different one for the next thousand. With a decaying learning rate, whatever comes last gets learned with the smallest steps and whatever comes first gets partially overwritten."}]},{"t":"code","lang":"python","label":"Python","text":"dataset = dataset.shuffle(seed=0)   # one line; not optional"},{"t":"p","runs":[{"text":"This is a real bug that produces a mysteriously bad model, and the loss curve looks fine throughout."}]},{"t":"h2","id":"multi-task-fine-tuning","runs":[{"text":"Multi-task fine-tuning"}]},{"t":"p","runs":[{"text":"If you have three tasks, one model trained on all three usually beats three separate adapters, provided the tasks are related. Shared structure transfers; the model learns your domain once rather than three times, and you maintain one artefact."}]},{"t":"p","runs":[{"text":"It stops being true when the tasks conflict: one wants terse structured output and another wants discursive explanation, and the model is asked to learn that the same kind of input maps to both. If your tasks pull in opposite directions, either separate them into different adapters or make the distinction explicit in the input: a system prompt or a tag that tells the model which mode it is in. Making it explicit is usually the better answer, because it keeps one artefact and gives you a controllable switch."}]},{"t":"h2","id":"the-rule-to-keep","runs":[{"text":"The rule to keep"}]},{"t":"p","runs":[{"text":"Before you train, write down the proportions of your file by source and by intent, as actual counts. Almost nobody does this, and almost every surprising fine-tune result is explained by it."}]}],"check":{"question":[{"text":"A team’s training file is ordered by source: 3,000 triage examples followed by 1,000 general conversations. They shuffle nothing and use a cosine schedule. Why is this worse than the same data shuffled?"}],"options":[{"text":[{"text":"Unshuffled data causes the optimiser’s momentum buffers to saturate, which prevents later examples from producing gradients."}],"correct":false,"tempting":[{"text":"Invents a saturation mechanism; momentum is an exponential average that adapts continuously and does not stop responding to new gradients."}]},{"text":[{"text":"The general examples arrive last, where the learning rate is smallest, so they influence the weights least."}],"correct":true,"tempting":null},{"text":[{"text":"Batches drawn from a single source have lower gradient variance, so the effective learning rate is higher and the run diverges."}],"correct":false,"tempting":[{"text":"Correctly notes within-batch homogeneity but draws the wrong conclusion; low variance does not raise the learning rate or cause divergence."}]},{"text":[{"text":"The model will overfit the 3,000 triage examples in the first epoch before ever seeing general data, and later data cannot undo memorisation."}],"correct":false,"tempting":[{"text":"Assumes memorisation is irreversible, when the real asymmetry is that the schedule gives the later data much smaller steps."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"distillation-datasets-in-practice","title":"Generating a dataset from a teacher","minutes":10},"next":{"slug":"packing-and-long-context","title":"Packing, document boundaries and long context","minutes":9}}}