{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"continued-pretraining","title":"When an adapter is not enough","url":"https://addaly.com/learn/running-models-yourself/continued-pretraining","position":81,"of":82,"minutes":9,"answer":"Continued pretraining is for distribution shifts an adapter cannot cross, needs hundreds of millions of in-domain tokens.","keyIdea":"Continued pretraining is for distribution shifts an adapter cannot cross, needs hundreds of millions of in-domain tokens, and strips instruction following so it must be followed by a second supervised stage. And for most projects the better move is finding a base model already trained for that language or domain."},"body":{"format":"markdown","text":"## The boundary LoRA cannot cross\n\nA LoRA adapter shifts behaviour within what the base already represents. Where the required change is large and broad (a language the model barely saw, a script it cannot tokenise efficiently, a technical domain whose vocabulary and statistics differ from the pretraining corpus) no affordable rank closes the gap. The symptom is recognisable: the adapter improves format and tone, and the underlying competence does not move.\n\nAt that point the remaining option is continued pretraining: taking the base model and training it further on raw text with the original objective, predicting the next token, with no instruction format at all.\n\n## What it costs\n\nThe scale is different in kind. Instruction tuning works on hundreds or thousands of examples. Continued pretraining works on hundreds of millions to billions of tokens, and the compute is proportionate: days to weeks on rented multi-GPU hardware for a model in the 7B class.\n\nWork out the token budget before anything else. A reasonable rule of thumb for meaningful domain adaptation is that you need a corpus at least in the hundreds of millions of tokens, and that corpus has to be in your domain rather than adjacent to it. If you have ten million tokens of internal documents, continued pretraining is not available to you, and retrieval is what you were looking for.\n\n## The failure that catches everyone\n\nContinued pretraining on raw text removes instruction following. The base you started from was instruction-tuned; you have just trained it on documents that contain no instructions and no assistant turns, and it reverts to completing text.\n\nThe model that comes out will continue your sentence rather than answer your question. This is not a bug in the run; it is what the objective asked for. The remedy is to re-apply instruction tuning afterwards, a supervised fine-tune on chat data, which is the earlier part of this module, and it means the project is two training stages rather than one.\n\nA common mitigation is to mix a small proportion of instruction-formatted data into the continued-pretraining corpus, on the order of a few per cent, which reduces how far the model drifts. It reduces rather than removes the need for the second stage.\n\n## The tokeniser problem\n\nFor a new language or script, look at tokenisation before you look at anything else:\n\n```python\nfor s in [\"The invoice is overdue.\", \"चालान बकाया है।\", \"இன்வாய்ஸ் நிலுவையில் உள்ளது.\"]:\n    print(len(tok.encode(s)), \"tokens for\", len(s), \"characters\")\n```\n\nIf a script comes out at three or four tokens per character, everything about that language is more expensive on this model: context is consumed several times faster, generation is several times slower per unit of meaning, and the cost per useful output is multiplied accordingly. This is a real and measurable inequality between languages, and it is a property of the tokeniser rather than of the model's knowledge.\n\nExtending the vocabulary is possible: add tokens, resize the embedding matrix, and train the new embeddings before training everything else. It is a genuine project with its own failure modes, and it makes the model incompatible with the original tokeniser and therefore with anything that assumed it.\n\nThe cheaper answer, almost always, is to find a base model that was already trained with that language properly represented. Several open model families are explicitly multilingual, and some are built for specific language groups. Checking for one costs an afternoon and frequently ends the project happily.\n\n## Use LoRA here too, with different settings\n\nContinued pretraining with adapters is possible and is much cheaper than full fine-tuning. It needs different settings from instruction tuning: a substantially higher rank, and the embedding and output layers included in the target modules, because new vocabulary and new domain statistics are exactly what those layers hold.\n\nIt still underperforms full continued pretraining for large distribution shifts, for the low-rank reason given earlier. Treat it as the version you can afford rather than the version that is equivalent.\n\n## The decision, plainly\n\nFor almost everyone reading this, continued pretraining is the wrong tool, and saying so is more useful than describing it well. The order to work through:\n\n1. Is there an existing open model already trained for this language or domain? Usually yes. Use it.\n2. Is the problem actually knowledge rather than capability? Then retrieval.\n3. Is the problem format, tone or task behaviour? Then a LoRA on a few hundred examples.\n4. Only if the base cannot represent the domain, and you have a corpus in the hundreds of millions of tokens and a budget for rented compute, does continued pretraining become the answer.\n\n> The strongest move in this area is usually choosing a different base model, not training the one you started with."},"check":{"question":"A team continues pretraining an instruction-tuned model on 400 million tokens of domain documents. The result has clearly absorbed the domain but now continues the user's sentence instead of answering. What happened?","options":[{"text":"The learning rate was too high, so the model's instruction-following weights were overwritten.","correct":false,"tempting":"Names a symptom-shaped cause; even at a careful rate, training on documents with no assistant turns moves the model towards completion."},{"text":"The domain corpus lacked question-and-answer pairs, so the model has no examples of the domain in dialogue form.","correct":false,"tempting":"Close, but it frames a general objective shift as a coverage gap; the model has lost instruction following on all topics, not only domain ones."},{"text":"The chat template was lost when the model was saved, so the serving stack is no longer wrapping messages correctly.","correct":false,"tempting":"A real and worth-checking failure with a similar surface, but it would not be accompanied by genuine absorption of domain content."},{"text":"Raw-text training has no instruction format, so it trains towards completion and needs a second stage.","correct":true}]},"prev":{"slug":"serving-and-merging-adapters","title":"Serving what you made, merged or not","url":"https://addaly.com/learn/running-models-yourself/serving-and-merging-adapters"},"next":{"slug":"what-you-may-ship","title":"What you may ship, and what nobody has settled","url":"https://addaly.com/learn/running-models-yourself/what-you-may-ship"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When an adapter is not enough', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/continued-pretraining (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When an adapter is not enough","courseTitle":"Running Models Yourself","module":{"index":9,"slug":"making-it-yours","title":"Making it yours: adapters and training"},"answer":null,"keyIdea":"Continued pretraining is for distribution shifts an adapter cannot cross, needs hundreds of millions of in-domain tokens, and strips instruction following so it must be followed by a second supervised stage. And for most projects the better move is finding a base model already trained for that language or domain.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-boundary-lora-cannot-cross","runs":[{"text":"The boundary LoRA cannot cross"}]},{"t":"p","runs":[{"text":"A LoRA adapter shifts behaviour within what the base already represents. Where the required change is large and broad (a language the model barely saw, a script it cannot tokenise efficiently, a technical domain whose vocabulary and statistics differ from the pretraining corpus) no affordable rank closes the gap. The symptom is recognisable: the adapter improves format and tone, and the underlying competence does not move."}]},{"t":"p","runs":[{"text":"At that point the remaining option is continued pretraining: taking the base model and training it further on raw text with the original objective, predicting the next token, with no instruction format at all."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"The scale is different in kind. Instruction tuning works on hundreds or thousands of examples. Continued pretraining works on hundreds of millions to billions of tokens, and the compute is proportionate: days to weeks on rented multi-GPU hardware for a model in the 7B class."}]},{"t":"p","runs":[{"text":"Work out the token budget before anything else. A reasonable rule of thumb for meaningful domain adaptation is that you need a corpus at least in the hundreds of millions of tokens, and that corpus has to be in your domain rather than adjacent to it. If you have ten million tokens of internal documents, continued pretraining is not available to you, and retrieval is what you were looking for."}]},{"t":"h2","id":"the-failure-that-catches-everyone","runs":[{"text":"The failure that catches everyone"}]},{"t":"p","runs":[{"text":"Continued pretraining on raw text removes instruction following. The base you started from was instruction-tuned; you have just trained it on documents that contain no instructions and no assistant turns, and it reverts to completing text."}]},{"t":"p","runs":[{"text":"The model that comes out will continue your sentence rather than answer your question. This is not a bug in the run; it is what the objective asked for. The remedy is to re-apply instruction tuning afterwards, a supervised fine-tune on chat data, which is the earlier part of this module, and it means the project is two training stages rather than one."}]},{"t":"p","runs":[{"text":"A common mitigation is to mix a small proportion of instruction-formatted data into the continued-pretraining corpus, on the order of a few per cent, which reduces how far the model drifts. It reduces rather than removes the need for the second stage."}]},{"t":"h2","id":"the-tokeniser-problem","runs":[{"text":"The tokeniser problem"}]},{"t":"p","runs":[{"text":"For a new language or script, look at tokenisation before you look at anything else:"}]},{"t":"code","lang":"python","label":"Python","text":"for s in [\"The invoice is overdue.\", \"चालान बकाया है।\", \"இன்வாய்ஸ் நிலுவையில் உள்ளது.\"]:\n    print(len(tok.encode(s)), \"tokens for\", len(s), \"characters\")"},{"t":"p","runs":[{"text":"If a script comes out at three or four tokens per character, everything about that language is more expensive on this model: context is consumed several times faster, generation is several times slower per unit of meaning, and the cost per useful output is multiplied accordingly. This is a real and measurable inequality between languages, and it is a property of the tokeniser rather than of the model’s knowledge."}]},{"t":"p","runs":[{"text":"Extending the vocabulary is possible: add tokens, resize the embedding matrix, and train the new embeddings before training everything else. It is a genuine project with its own failure modes, and it makes the model incompatible with the original tokeniser and therefore with anything that assumed it."}]},{"t":"p","runs":[{"text":"The cheaper answer, almost always, is to find a base model that was already trained with that language properly represented. Several open model families are explicitly multilingual, and some are built for specific language groups. Checking for one costs an afternoon and frequently ends the project happily."}]},{"t":"h2","id":"use-lora-here-too-with-different-settings","runs":[{"text":"Use LoRA here too, with different settings"}]},{"t":"p","runs":[{"text":"Continued pretraining with adapters is possible and is much cheaper than full fine-tuning. It needs different settings from instruction tuning: a substantially higher rank, and the embedding and output layers included in the target modules, because new vocabulary and new domain statistics are exactly what those layers hold."}]},{"t":"p","runs":[{"text":"It still underperforms full continued pretraining for large distribution shifts, for the low-rank reason given earlier. Treat it as the version you can afford rather than the version that is equivalent."}]},{"t":"h2","id":"the-decision-plainly","runs":[{"text":"The decision, plainly"}]},{"t":"p","runs":[{"text":"For almost everyone reading this, continued pretraining is the wrong tool, and saying so is more useful than describing it well. The order to work through:"}]},{"t":"ol","start":1,"items":[[{"text":"Is there an existing open model already trained for this language or domain? Usually yes. Use it."}],[{"text":"Is the problem actually knowledge rather than capability? Then retrieval."}],[{"text":"Is the problem format, tone or task behaviour? Then a LoRA on a few hundred examples."}],[{"text":"Only if the base cannot represent the domain, and you have a corpus in the hundreds of millions of tokens and a budget for rented compute, does continued pretraining become the answer."}]]},{"t":"quote","runs":[{"text":"The strongest move in this area is usually choosing a different base model, not training the one you started with."}]}],"check":{"question":[{"text":"A team continues pretraining an instruction-tuned model on 400 million tokens of domain documents. The result has clearly absorbed the domain but now continues the user’s sentence instead of answering. What happened?"}],"options":[{"text":[{"text":"The learning rate was too high, so the model’s instruction-following weights were overwritten."}],"correct":false,"tempting":[{"text":"Names a symptom-shaped cause; even at a careful rate, training on documents with no assistant turns moves the model towards completion."}]},{"text":[{"text":"The domain corpus lacked question-and-answer pairs, so the model has no examples of the domain in dialogue form."}],"correct":false,"tempting":[{"text":"Close, but it frames a general objective shift as a coverage gap; the model has lost instruction following on all topics, not only domain ones."}]},{"text":[{"text":"The chat template was lost when the model was saved, so the serving stack is no longer wrapping messages correctly."}],"correct":false,"tempting":[{"text":"A real and worth-checking failure with a similar surface, but it would not be accompanied by genuine absorption of domain content."}]},{"text":[{"text":"Raw-text training has no instruction format, so it trains towards completion and needs a second stage."}],"correct":true,"tempting":null}],"order":[1,0,2,3],"why":null},"prev":{"slug":"serving-and-merging-adapters","title":"Serving what you made, merged or not","minutes":8},"next":{"slug":"what-you-may-ship","title":"What you may ship, and what nobody has settled","minutes":9}}}