What to log, and how to read it

Loss is one of six numbers

Most people watch the loss and nothing else. Here is the full set worth having, and what each one tells you that loss does not.

Gradient norm. The most under-used signal in fine-tuning. It is the magnitude of the whole gradient vector before clipping, and the shape of its curve is diagnostic:

• A healthy run: moderate at the start, settling to a stable band.
• Isolated tall spikes: individual pathological examples. Harmless if max_grad_norm is clipping them, and worth finding anyway: log the batch index.
• Steadily climbing: instability. The learning rate is too high and the run is heading for divergence, and you can see it several hundred steps before the loss does.
• Near zero from the start: nothing is learning. Check trainable parameters before anything else.

Tokens per optimiser step. Not logged by default, and it is the true unit of batch size. Compute it from the attention mask and log it once at the start; if it is under a few thousand, your gradients are noisy regardless of what the example count says.

Token accuracy on unmasked positions. The fraction of target positions where the argmax is correct. Free to compute, interpretable without exponentiating anything, and less swayed than loss by a handful of very surprising tokens. When loss and token accuracy disagree, a small number of examples are dominating the loss, and finding them usually finds a data problem.

Learning rate. Log it, because a schedule that is not doing what you think (no warmup, a decay set for the wrong number of steps) is invisible otherwise and completely obvious on a plot.

Evaluation loss. Every hundred steps or so on a held-out slice, to see the training–validation gap open.

Mean generated length. Generate on a few fixed prompts at each evaluation and record the mean token count. This is the cheapest detector of the length drift that preference methods cause and that model judges reward.

The samples

Three fixed prompts, generated at every evaluation, logged as text.

This is not a metric and it is more informative than all of them. A loss curve cannot show you that the model has started every answer with the same clause, dropped its bullet points, begun repeating itself at the end of long answers, or lost its British spelling. Three samples show all of that at a glance.

Pick the three deliberately: one typical input, one edge case, one input that should be declined.

The tools, all free

TensorBoard is the plain answer. It is included with PyTorch, runs entirely locally, needs no account, and handles scalars and text. report_to="tensorboard" and tensorboard --logdir out/.

Trackio is a newer free, open-source tracker with a Weights & Biases-compatible API, runnable locally or hosted on a Hugging Face Space at no cost. Useful when you want a shareable dashboard without a commercial account.

MLflow and Aim are free and self-hostable, with more experiment-management machinery than a single fine-tune needs but real value once you have twenty runs to compare.

Weights & Biases is the commercial standard with a free personal tier. Convenient, and it sends your run metadata to a third party, which for a project training on customer data is a question to answer before the first run rather than after.

Estimating the run at step 20

# after 20 steps
sec_per_step = elapsed / 20
total_hours  = sec_per_step * total_steps / 3600
print(f"{sec_per_step:.2f}s/step → {total_hours:.1f}h → ${total_hours*0.60:.2f}")

Run this twenty steps in. It tells you whether your six-hour plan is a six-hour plan or a fifty-hour one, while cancelling is still free. Throughput per step is roughly constant after warmup, so the extrapolation is reliable.

What to keep after the run

The TensorBoard directory is a few megabytes. Keep it beside the checkpoint, in the same place, forever. Three months later, when somebody asks why version 2 is better than version 3, the answer is in those curves; and if they were only in a terminal you closed, the answer is a shrug.