Watching a run you cannot sit next to

What to log, and why the default is not enough

A training run prints a loss every logging_steps. That one number tells you whether the run is alive and almost nothing else. Four things are worth having:

1. Training loss: is it falling.
2. Validation loss or metric: is it generalising, which is a different question.
3. Learning rate: did warmup and decay behave as configured.
4. Gradient norm: spikes here precede a divergence you can otherwise only see after it has happened.

All four are logged automatically with report_to set:

TrainingArguments(
    ...,
    logging_steps=10,
    eval_strategy="steps", eval_steps=100,
    report_to="tensorboard",      # or "trackio", "wandb", "none"
)

Where to put the logs

TensorBoard writes event files into output_dir. If push_to_hub=True is set, those files go to the model repository and the Hub renders them on a Training metrics tab, so a stranger reading your model card can see the loss curve. Free, no account, no service.

Trackio is a lightweight open-source tracker that runs as a Space, storing runs in a dataset repository. It is free, self-hosted and the API mirrors the commercial trackers closely enough that switching is a one-line change.

Weights & Biases and similar services are free for personal use and excellent, with the ordinary caveat that your run metadata goes to a third party. On a work project that may need asking about first.

The important thing is that the record survives the machine. A loss curve that existed only in a Colab output cell is gone when the session ends, and with it the ability to compare this run to the next one.

Reading a loss curve

Four shapes, and what each means:

• Both losses fall and level off. Working. Stop when validation stops improving.
• Training falls, validation rises. Overfitting. Fewer epochs, more data, more dropout, or a lower LoRA rank. load_best_model_at_end keeps the checkpoint from before the turn.
• Both flat. Learning rate too low, or the labels are not reaching the loss: check the masking. A completely flat loss from step one is nearly always a data bug, not a learning-rate problem.
• Loss spikes to NaN. Learning rate too high, or float16 overflow. Lower the rate; switch to bf16.

A jagged curve is normal at small batch sizes. A curve that is jagged and trending flat usually means the effective batch is too small: raise gradient accumulation before you change anything else.

Checkpoints, and how not to lose a night

save_strategy="steps", save_steps=100,
save_total_limit=2,
load_best_model_at_end=True,
push_to_hub=True, hub_strategy="checkpoint",

hub_strategy="checkpoint" pushes the checkpoint to the Hub as it saves, so the run survives the machine dying. save_total_limit=2 keeps the disk from filling: each checkpoint is a full copy of the trainable state, and on a free instance three of them can be the thing that kills the session.

Resuming is one argument:

trainer.train(resume_from_checkpoint=True)

This restores optimiser state and the scheduler position as well as weights. Restarting from a checkpoint by loading only the weights and beginning again resets the learning-rate schedule, which produces a visible jump in the loss and a worse model. Use the argument.

Seeds and what reproducibility means here

seed=42, data_seed=42

Those make the run repeatable on the same hardware with the same library versions. Across different GPUs or versions, floating-point accumulation order differs and results drift slightly. That is expected; claiming bit-identical reproducibility across machines is a claim nobody can keep.

What you can keep: log the versions, the seed, the base model revision and the dataset revision into the run. Then a result is reconstructible even when it is not bit-identical, which is what anybody actually needs.

Evaluate the model, not the loss

Validation loss going down is necessary and not sufficient. Before declaring a run successful, generate output for twenty held-out inputs and read them. Loss can improve while the model develops a habit you would immediately reject: always answering in English, always producing three bullet points, losing the ability to say it does not know.

This is twenty minutes and it is the only check that catches that class of problem.

Do this now

Set report_to="tensorboard" and push_to_hub=True on your next run, then open the Training metrics tab on the resulting repository. A loss curve attached to the model it produced is a small thing that makes every later comparison possible.