{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"continued-pretraining","title":"Continued pretraining on raw text","url":"https://addaly.com/learn/fine-tuning/continued-pretraining","position":46,"of":84,"minutes":9,"answer":"Continued pretraining puts loss on every token of raw text to install a domain's register and vocabulary, needs hundreds of millions of tokens rather than thousands of examples, and reliably removes instruction-following.","keyIdea":"Continued pretraining puts loss on every token of raw text to install a domain's register and vocabulary, needs hundreds of millions of tokens rather than thousands of examples, and reliably removes instruction-following, so it is always followed by redoing SFT, and it still will not make the model reliable about any specific document."},"body":{"format":"markdown","text":"## A different objective on different data\n\nEverything else in this course trains on instruction-shaped examples with the prompt masked. Continued pretraining does neither. It trains on raw text (documents, transcripts, code, books) with loss on every token, exactly as the original pretraining did.\n\nThe purpose is different too. Instruction tuning teaches a model how to behave. Continued pretraining teaches it a domain: the vocabulary, the register, the statistical texture of a body of writing it has seen little of.\n\n## When it is the right tool\n\nFour situations, and they are recognisable.\n\n**A language the base model barely saw.** Not \"it answers in Marathi somewhat clumsily\". That is instruction tuning. This is \"the model's Marathi is not fluent, and its tokenisation of Devanagari is producing four tokens per word\". Representations are absent; no adapter over instruction data installs them.\n\n**A specialised register.** Clinical notes with their abbreviations, legal drafting, chip design, a scientific subfield with its own notation. If a domain expert reads the base model's output and says the words are right but nobody writes like that, the texture is missing.\n\n**A proprietary corpus.** Twenty years of internal engineering reports, where the value is in the accumulated conventions rather than in any single fact. Note carefully: this is not the way to make the model *know* your documents. That is retrieval. This is the way to make it *write like* them.\n\n**A code ecosystem the model does not know.** An internal framework, a rare language, a DSL.\n\n## The volume, and it is large\n\nThis is where continued pretraining separates from everything else in the course. Instruction tuning works with thousands of examples. Domain adaptation works with tokens, by the hundred million:\n\n- **A light register shift**: 100 million to 1 billion tokens.\n- **A genuine domain**: 1 to 10 billion tokens.\n- **A new language**: 10 billion and up, typically with a tokenizer change as well.\n\nA billion tokens is roughly 4 GB of text: perhaps two million documents. If you have 50,000 internal documents, you have something like 50 million tokens, which is enough to shift register slightly and not enough to teach a domain. Knowing that before you start saves a month.\n\n## Forgetting, at full strength\n\nThe most predictable outcome of continued pretraining is that the model stops following instructions. You have just trained it, on raw text, that its job is to continue documents. The instruction-following behaviour that made it useful came from a later stage you have now partly undone.\n\nThis is expected, and the pipeline accounts for it:\n\n```\ncontinued pretraining  →  SFT again  →  preference tuning (optional)\n```\n\nYou redo instruction tuning afterwards. Plan for it, budget for it, and keep the SFT dataset from before, because you will need it.\n\nThe second mitigation is replay, at higher proportions than elsewhere: mix 10 to 30% of general-purpose text into your domain corpus. Without it, a model trained on 100% clinical notes becomes a model that can only produce clinical notes.\n\n## Configuration that differs from SFT\n\n- **No masking.** Loss on every token; the whole document is the target.\n- **Pack aggressively.** Documents are long and variable; packing with correct document separation is standard here, not optional.\n- **Lower learning rate** than instruction tuning if you are doing it fully: 1e-5 or below. You are nudging a pretrained model, and pretraining-scale damage is easy to do.\n- **Higher LoRA rank if using an adapter**: 128 or 256, and include the embedding and output layers via `modules_to_save`. A rank-16 adapter is a reweighting, and domain adaptation is not a reweighting. Many practitioners do this stage fully rather than with an adapter, for that reason.\n- **One epoch.** With hundreds of millions of tokens you do not need a second pass, and a second pass is where memorisation lives.\n\n## The cheaper alternative to try first\n\nBefore committing to a billion tokens, test the hypothesis directly. Take 200 domain questions. Answer them with the base model given a few in-domain examples in the prompt, and with retrieval over your corpus. If either gets you most of the way, the domain gap was smaller than it felt.\n\nAnd if the actual complaint is that the model does not know specific facts from your documents, continued pretraining is the wrong tool, expensively. It will improve how the model writes about your domain and it will not make it reliable about what is in any particular document. Facts belong in the context window. That rule has not changed for being expensive to ignore."},"check":{"question":"A team continued-pretrains a chat model on 400 million tokens of internal engineering reports. The output now reads convincingly like their reports, but the model no longer follows instructions properly. What happened, and what is the fix?","options":[{"text":"The learning rate was too high, destroying the instruction-tuned weights; rerunning at a lower rate would preserve both.","correct":false,"tempting":"Treats an objective mismatch as a magnitude problem; a lower rate slows the drift but does not change what the objective is teaching."},{"text":"Unmasked raw text taught the model to continue documents rather than answer, so SFT must be redone.","correct":true},{"text":"Four hundred million tokens is far too many for a chat model, which should be adapted with at most a few million.","correct":false,"tempting":"Inverts the volume guidance; 400 million is modest for domain adaptation and the forgetting would occur at any scale."},{"text":"The reports contain no dialogue, so the model lost its chat template; reapplying the template at inference restores the behaviour.","correct":false,"tempting":"Reduces a behavioural change to a formatting one, when the template is applied at inference regardless and the model still does not follow instructions."}]},"prev":{"slug":"reward-hacking","title":"Reward hacking","url":"https://addaly.com/learn/fine-tuning/reward-hacking"},"next":{"slug":"heads-instead-of-tokens","title":"Training a head instead of generating tokens","url":"https://addaly.com/learn/fine-tuning/heads-instead-of-tokens"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Continued pretraining on raw text', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/continued-pretraining (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Continued pretraining on raw text","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"Continued pretraining puts loss on every token of raw text to install a domain’s register and vocabulary, needs hundreds of millions of tokens rather than thousands of examples, and reliably removes instruction-following, so it is always followed by redoing SFT, and it still will not make the model reliable about any specific document.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-different-objective-on-different-data","runs":[{"text":"A different objective on different data"}]},{"t":"p","runs":[{"text":"Everything else in this course trains on instruction-shaped examples with the prompt masked. Continued pretraining does neither. It trains on raw text (documents, transcripts, code, books) with loss on every token, exactly as the original pretraining did."}]},{"t":"p","runs":[{"text":"The purpose is different too. Instruction tuning teaches a model how to behave. Continued pretraining teaches it a domain: the vocabulary, the register, the statistical texture of a body of writing it has seen little of."}]},{"t":"h2","id":"when-it-is-the-right-tool","runs":[{"text":"When it is the right tool"}]},{"t":"p","runs":[{"text":"Four situations, and they are recognisable."}]},{"t":"p","runs":[{"text":"A language the base model barely saw.","b":true},{"text":" Not “it answers in Marathi somewhat clumsily”. That is instruction tuning. This is “the model’s Marathi is not fluent, and its tokenisation of Devanagari is producing four tokens per word”. Representations are absent; no adapter over instruction data installs them."}]},{"t":"p","runs":[{"text":"A specialised register.","b":true},{"text":" Clinical notes with their abbreviations, legal drafting, chip design, a scientific subfield with its own notation. If a domain expert reads the base model’s output and says the words are right but nobody writes like that, the texture is missing."}]},{"t":"p","runs":[{"text":"A proprietary corpus.","b":true},{"text":" Twenty years of internal engineering reports, where the value is in the accumulated conventions rather than in any single fact. Note carefully: this is not the way to make the model "},{"text":"know","i":true},{"text":" your documents. That is retrieval. This is the way to make it "},{"text":"write like","i":true},{"text":" them."}]},{"t":"p","runs":[{"text":"A code ecosystem the model does not know.","b":true},{"text":" An internal framework, a rare language, a DSL."}]},{"t":"h2","id":"the-volume-and-it-is-large","runs":[{"text":"The volume, and it is large"}]},{"t":"p","runs":[{"text":"This is where continued pretraining separates from everything else in the course. Instruction tuning works with thousands of examples. Domain adaptation works with tokens, by the hundred million:"}]},{"t":"ul","items":[[{"text":"A light register shift","b":true},{"text":": 100 million to 1 billion tokens."}],[{"text":"A genuine domain","b":true},{"text":": 1 to 10 billion tokens."}],[{"text":"A new language","b":true},{"text":": 10 billion and up, typically with a tokenizer change as well."}]]},{"t":"p","runs":[{"text":"A billion tokens is roughly 4 GB of text: perhaps two million documents. If you have 50,000 internal documents, you have something like 50 million tokens, which is enough to shift register slightly and not enough to teach a domain. Knowing that before you start saves a month."}]},{"t":"h2","id":"forgetting-at-full-strength","runs":[{"text":"Forgetting, at full strength"}]},{"t":"p","runs":[{"text":"The most predictable outcome of continued pretraining is that the model stops following instructions. You have just trained it, on raw text, that its job is to continue documents. The instruction-following behaviour that made it useful came from a later stage you have now partly undone."}]},{"t":"p","runs":[{"text":"This is expected, and the pipeline accounts for it:"}]},{"t":"code","lang":"","label":"Text","text":"continued pretraining  →  SFT again  →  preference tuning (optional)"},{"t":"p","runs":[{"text":"You redo instruction tuning afterwards. Plan for it, budget for it, and keep the SFT dataset from before, because you will need it."}]},{"t":"p","runs":[{"text":"The second mitigation is replay, at higher proportions than elsewhere: mix 10 to 30% of general-purpose text into your domain corpus. Without it, a model trained on 100% clinical notes becomes a model that can only produce clinical notes."}]},{"t":"h2","id":"configuration-that-differs-from-sft","runs":[{"text":"Configuration that differs from SFT"}]},{"t":"ul","items":[[{"text":"No masking.","b":true},{"text":" Loss on every token; the whole document is the target."}],[{"text":"Pack aggressively.","b":true},{"text":" Documents are long and variable; packing with correct document separation is standard here, not optional."}],[{"text":"Lower learning rate","b":true},{"text":" than instruction tuning if you are doing it fully: 1e-5 or below. You are nudging a pretrained model, and pretraining-scale damage is easy to do."}],[{"text":"Higher LoRA rank if using an adapter","b":true},{"text":": 128 or 256, and include the embedding and output layers via "},{"text":"modules_to_save","code":true},{"text":". A rank-16 adapter is a reweighting, and domain adaptation is not a reweighting. Many practitioners do this stage fully rather than with an adapter, for that reason."}],[{"text":"One epoch.","b":true},{"text":" With hundreds of millions of tokens you do not need a second pass, and a second pass is where memorisation lives."}]]},{"t":"h2","id":"the-cheaper-alternative-to-try-first","runs":[{"text":"The cheaper alternative to try first"}]},{"t":"p","runs":[{"text":"Before committing to a billion tokens, test the hypothesis directly. Take 200 domain questions. Answer them with the base model given a few in-domain examples in the prompt, and with retrieval over your corpus. If either gets you most of the way, the domain gap was smaller than it felt."}]},{"t":"p","runs":[{"text":"And if the actual complaint is that the model does not know specific facts from your documents, continued pretraining is the wrong tool, expensively. It will improve how the model writes about your domain and it will not make it reliable about what is in any particular document. Facts belong in the context window. That rule has not changed for being expensive to ignore."}]}],"check":{"question":[{"text":"A team continued-pretrains a chat model on 400 million tokens of internal engineering reports. The output now reads convincingly like their reports, but the model no longer follows instructions properly. What happened, and what is the fix?"}],"options":[{"text":[{"text":"The learning rate was too high, destroying the instruction-tuned weights; rerunning at a lower rate would preserve both."}],"correct":false,"tempting":[{"text":"Treats an objective mismatch as a magnitude problem; a lower rate slows the drift but does not change what the objective is teaching."}]},{"text":[{"text":"Unmasked raw text taught the model to continue documents rather than answer, so SFT must be redone."}],"correct":true,"tempting":null},{"text":[{"text":"Four hundred million tokens is far too many for a chat model, which should be adapted with at most a few million."}],"correct":false,"tempting":[{"text":"Inverts the volume guidance; 400 million is modest for domain adaptation and the forgetting would occur at any scale."}]},{"text":[{"text":"The reports contain no dialogue, so the model lost its chat template; reapplying the template at inference restores the behaviour."}],"correct":false,"tempting":[{"text":"Reduces a behavioural change to a formatting one, when the template is applied at inference regardless and the model still does not follow instructions."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"reward-hacking","title":"Reward hacking","minutes":9},"next":{"slug":"heads-instead-of-tokens","title":"Training a head instead of generating tokens","minutes":9}}}