When the model does not have your language at all

A different scale of intervention

Everything so far assumed the model already had the underlying capability and needed its behaviour shaped. Continued pretraining is for when it does not: a language it barely saw, a script it cannot tokenise, a domain with notation it has never encountered.

This is training, not tuning. It uses raw text, not instruction pairs, with the ordinary next-token objective, and it needs volume:

• A domain: legal filings, clinical notes, a company’s technical corpus: hundreds of millions of tokens is a meaningful run.
• A language the model saw a little of: billions of tokens.
• A language the model has essentially never seen: tens of billions, and you are approaching the cost of training a small model from scratch.

For scale, an 8B model processing 10 billion tokens is in the region of a thousand GPU-hours on a datacentre card. That is a real budget, a real timeline and a real risk of a failed run, and it is why this is the least commonly justified item in this block.

Vocabulary extension

If the target language tokenises badly (the fertility measurement from an earlier lesson showing five or more tokens per word) you may need to add tokens before training.

The procedure: train a tokenizer on your corpus, take the merges the base tokenizer lacks, extend the vocabulary, and resize the embedding and output matrices. The new rows need sensible initialisation, and the technique that works is to initialise each new token’s embedding as the mean of the embeddings of the pieces it used to be split into. Random initialisation makes the early training unstable and wastes a large amount of compute recovering.

Then train. Untrained new embeddings are worse than useless until they have seen enough text.

The step everybody forgets

Continued pretraining on raw text destroys instruction following. The model becomes a document continuer again, because that is what you just trained it to be. Ask the result a question and it produces more questions.

So the pipeline is three stages, and skipping the third produces a model people report as broken:

1. Continued pretraining on raw text in the target language or domain.
2. Supervised fine-tuning on instruction data, ideally in the same language, which usually means translating or writing an instruction set.
3. Optionally, preference tuning.

Stage 2 needs instruction data in the target language, and for underserved languages that data may not exist. Building it is often the larger part of the whole project.

The honest recommendation

For almost every team, the answer is not to do this.

The alternatives are better nearly always:

• Use a model that already has the language. The regional families from the earlier lesson exist precisely because organisations with the right corpora and the right budget already did this work.
• Fine-tune for register and format on a few hundred examples, which fixes what usually looks like a language problem and is a day of work.
• Retrieve in the target language, which fixes what is actually a knowledge problem.
• Use a translation stage, accepting its costs, if the output is machine-consumed rather than read.

Continued pretraining is justified when you have a underserved language, a corpus in the billions of tokens, the compute, and a reason nobody else will do it, which describes a research group, a national initiative or a well-funded regional effort, and very few product teams.

If you are in that position, the good news is that the recipe is public: AI2’s OLMo releases their data, code and checkpoints, and several regional projects have published their continued-pretraining pipelines in enough detail to follow. That is the practical value of the fully-open end of the openness spectrum, and this is the case where it matters most.