When an adapter is not enough

The boundary LoRA cannot cross

A LoRA adapter shifts behaviour within what the base already represents. Where the required change is large and broad (a language the model barely saw, a script it cannot tokenise efficiently, a technical domain whose vocabulary and statistics differ from the pretraining corpus) no affordable rank closes the gap. The symptom is recognisable: the adapter improves format and tone, and the underlying competence does not move.

At that point the remaining option is continued pretraining: taking the base model and training it further on raw text with the original objective, predicting the next token, with no instruction format at all.

What it costs

The scale is different in kind. Instruction tuning works on hundreds or thousands of examples. Continued pretraining works on hundreds of millions to billions of tokens, and the compute is proportionate: days to weeks on rented multi-GPU hardware for a model in the 7B class.

Work out the token budget before anything else. A reasonable rule of thumb for meaningful domain adaptation is that you need a corpus at least in the hundreds of millions of tokens, and that corpus has to be in your domain rather than adjacent to it. If you have ten million tokens of internal documents, continued pretraining is not available to you, and retrieval is what you were looking for.

The failure that catches everyone

Continued pretraining on raw text removes instruction following. The base you started from was instruction-tuned; you have just trained it on documents that contain no instructions and no assistant turns, and it reverts to completing text.

The model that comes out will continue your sentence rather than answer your question. This is not a bug in the run; it is what the objective asked for. The remedy is to re-apply instruction tuning afterwards, a supervised fine-tune on chat data, which is the earlier part of this module, and it means the project is two training stages rather than one.

A common mitigation is to mix a small proportion of instruction-formatted data into the continued-pretraining corpus, on the order of a few per cent, which reduces how far the model drifts. It reduces rather than removes the need for the second stage.

The tokeniser problem

For a new language or script, look at tokenisation before you look at anything else:

for s in ["The invoice is overdue.", "चालान बकाया है।", "இன்வாய்ஸ் நிலுவையில் உள்ளது."]:
    print(len(tok.encode(s)), "tokens for", len(s), "characters")

If a script comes out at three or four tokens per character, everything about that language is more expensive on this model: context is consumed several times faster, generation is several times slower per unit of meaning, and the cost per useful output is multiplied accordingly. This is a real and measurable inequality between languages, and it is a property of the tokeniser rather than of the model’s knowledge.

Extending the vocabulary is possible: add tokens, resize the embedding matrix, and train the new embeddings before training everything else. It is a genuine project with its own failure modes, and it makes the model incompatible with the original tokeniser and therefore with anything that assumed it.

The cheaper answer, almost always, is to find a base model that was already trained with that language properly represented. Several open model families are explicitly multilingual, and some are built for specific language groups. Checking for one costs an afternoon and frequently ends the project happily.

Use LoRA here too, with different settings

Continued pretraining with adapters is possible and is much cheaper than full fine-tuning. It needs different settings from instruction tuning: a substantially higher rank, and the embedding and output layers included in the target modules, because new vocabulary and new domain statistics are exactly what those layers hold.

It still underperforms full continued pretraining for large distribution shifts, for the low-rank reason given earlier. Treat it as the version you can afford rather than the version that is equivalent.

The decision, plainly

For almost everyone reading this, continued pretraining is the wrong tool, and saying so is more useful than describing it well. The order to work through:

1. Is there an existing open model already trained for this language or domain? Usually yes. Use it.
2. Is the problem actually knowledge rather than capability? Then retrieval.
3. Is the problem format, tone or task behaviour? Then a LoRA on a few hundred examples.
4. Only if the base cannot represent the domain, and you have a corpus in the hundreds of millions of tokens and a budget for rented compute, does continued pretraining become the answer.

The strongest move in this area is usually choosing a different base model, not training the one you started with.