Debugging a run

The four checks to run before every job

These take under a minute together and catch the majority of fine-tuning bugs.

1. Print one templated example.

print(tok.apply_chat_template(train[0]["messages"], tokenize=False))

Read it. Every special token where you expect it, roles marked correctly, exactly one end-of-turn marker at the end.

2. Decode the unmasked labels of one real batch.

b = next(iter(trainer.get_train_dataloader()))
lbl = b["labels"][0]
print(tok.decode([t for t in lbl if t != -100]))

This should print exactly the assistant’s answer: not the prompt, not the system message, not nothing. If it prints the whole conversation, your masking is off. If it prints nothing, your response template string does not match. If it prints the answer without a trailing end token, you have the pad-equals-eos problem.

This single check is the highest-value five seconds in applied fine-tuning.

3. Print the trainable parameter count.

model.print_trainable_parameters()

Zero means your target modules do not exist in this architecture. A number equal to the full model means your adapter was not applied.

4. Print the first three token ids. One BOS, not two.

The overfit test

Before a real run, take eight examples and train on them for 100 steps with everything else identical.

The loss should fall to near zero. The model should reproduce those eight answers nearly verbatim.

If it cannot memorise eight examples, your pipeline is broken and no amount of data or hyperparameter tuning will help. This test costs two minutes and it separates “my data is not good enough” from “my code does not work”, which are otherwise indistinguishable from the outside and are debugged in completely different ways.

Symptom to mechanism

Loss is exactly 0.0. Every label is -100. The response template did not match; nothing is being trained on.

Loss is flat and non-zero. Learning rate far too low, no trainable parameters, or reentrant gradient checkpointing blocking gradients to the adapter. Check the parameter count first.

Loss drops to near zero within 50 steps. The answer is visible in the input. Either labels were not shifted, or the completion also appears in the prompt, which happens when a log export includes the response in a context field.

Loss is NaN. In fp16, overflow: lower the learning rate, confirm warmup, check for a corrupted example. In bf16, NaN is much rarer and usually means bad data: an empty string, an enormous unicode blob. Find it by logging the index of the batch where it first appears.

Loss spikes then recovers. Usually one pathological example. max_grad_norm=1.0 limits the damage. If it recurs at the same step each epoch, find that example and read it.

Out of memory at step 40, not step 1. Your longest sequences are later in the file. Either sort by length to fail fast, or lower max_length. With group_by_length=True the long batches cluster, which makes this both more likely and more predictable.

The model never stops generating. Missing end-of-sequence in training. Cause: pad equals eos and masking by id, or truncation cutting the end token off long examples, or packing without separators.

Generations are empty. The model emits end-of-sequence immediately. Usually the reverse masking error: trained on prompts rather than completions.

Evaluation looks excellent, production does not. Leakage between your training and evaluation sets, or a formatting difference between how you evaluate and how you serve. Check the second first: print the exact string sent in both paths and diff them.

Look at generations, not only at loss

Every two hundred steps, generate on three fixed prompts and log the output.

class Samples(TrainerCallback):
    def on_evaluate(self, args, state, control, model=None, **kw):
        for p in FIXED_PROMPTS:
            print(state.global_step, generate(model, p)[:200])

Loss curves hide everything that matters about text. A model that has started repeating itself, dropped its formatting, or begun every answer with the same clause shows all of that in three samples and none of it in the loss.

The discipline

Change one thing per run. Write down what you changed and what happened, in a file, in the repository. Four runs into a debugging session nobody remembers which combination produced the good result, and the log is what turns a frustrating afternoon into a result you can act on.