Loss is NaN, memory is gone, nothing is learning

An ordered list, because guessing is expensive

Training failures are slower to diagnose than inference failures, because each attempt costs an hour. Work down this list rather than trying things.

Out of memory

The most common failure and the most mechanical to fix. In order, because each step costs progressively more:

1. Reduce batch size, raise gradient accumulation. Free.
2. gradient_checkpointing=True. Costs 20 to 30% speed, saves a lot.
3. optim="adamw_bnb_8bit". Saves about three quarters of the optimiser state.
4. Shorten max_length. Check your actual length distribution first: often the 95th percentile is far below what you set.
5. LoRA, then QLoRA. The large wins.

One detail that catches people: out of memory during evaluation, not training. Evaluation batches default to the training batch size and generation builds a KV cache, so the peak can be higher than in training. Set per_device_eval_batch_size explicitly, lower.

And on a shared or notebook environment, memory from a previous failed run is often still held:

import gc, torch
del trainer, model
gc.collect(); torch.cuda.empty_cache()

If that does not free it, restart the kernel. Half an hour spent tuning batch size against a leak is half an hour wasted.

Loss is NaN

Three causes, in order of likelihood:

1. Learning rate too high. Drop it tenfold and see.
2. fp16 overflow. Switch to bf16 if the hardware allows. This is the T5-family failure from module three, appearing during training instead of inference.
3. Bad data. An empty string, a row whose labels are all -100 so the loss is dividing by zero, or a label index outside the model’s range. Check:

bad = [i for i, r in enumerate(ds) if all(l == -100 for l in r["labels"])]
print("rows with no supervised tokens:", len(bad))

A row with no unmasked labels is a real and common bug in instruction data: it happens when truncation cuts off the whole assistant turn.

Loss is completely flat from step one

This is almost never the learning rate. It is the labels not reaching the loss:

• The column is named label rather than labels.
• Every label position is masked to -100.
• With PEFT, no parameters are actually trainable: print_trainable_parameters() reporting zero means the config’s target_modules matched nothing, usually because the architecture’s module names differ from the ones you copied.
• The model was loaded with torch.no_grad() somewhere, or inputs do not require gradients under a quantized base, which is what prepare_model_for_kbit_training fixes.

Training loss falls beautifully, the model is useless

The worst case, and it has three usual causes:

The model learned the prompt. Prompt tokens were not masked, so it is being graded on reproducing the questions. Decode the unmasked labels and look.

Overfitting. Validation loss will show it. Fewer epochs, more data, lower rank.

The task was learned trivially. If every example in one class begins with the same phrase, the model learns the phrase. This is the data-quality lesson arriving late and expensive.

It works in training and not at inference

Almost always a mismatch between how examples were assembled for training and how prompts are assembled at inference:

• A different chat template, or none.
• add_generation_prompt=True missing at inference.
• A system message present during training and absent afterwards.
• Different truncation, so the model sees a shape it never trained on.

The test: render one training example and one inference prompt as strings and diff them. They should be identical up to the assistant’s turn. This finds the problem in two minutes and people spend days not doing it.

Catastrophic forgetting

The model gets better at your task and worse at everything else. Expected when full fine-tuning at a high learning rate. Mitigations, roughly in order: use LoRA rather than full fine-tuning, lower the learning rate, fewer epochs, and mix in some general instruction data alongside your task data.

Worth measuring rather than assuming: keep twenty general prompts unrelated to your task and run them before and after. If the answers have degraded, you have quantified the trade rather than discovered it in production.

The cheapest habit in this module

Overfit ten examples deliberately. Train on ten rows for many epochs and check that the model reproduces them almost exactly. If it cannot memorise ten examples, your pipeline is broken (labels, masking, trainable parameters, something) and no amount of data will help. Ten minutes, and it separates a pipeline bug from a data problem before you spend three hours on the wrong one.

Do this now

Run the ten-example overfit test on whatever you are about to train. If it works, your plumbing is sound and every later failure is about data or hyperparameters. That is a useful thing to know before hour three.