{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"logging-and-watching","title":"What to log, and how to read it","url":"https://addaly.com/learn/fine-tuning/logging-and-watching","position":54,"of":84,"minutes":8,"answer":"Gradient norm, tokens per step, token accuracy and three generated samples tell you things loss cannot: a run heading for divergence is visible in the gradient norm hundreds of steps before the loss moves.","keyIdea":"Gradient norm, tokens per step, token accuracy and three generated samples tell you things loss cannot: a run heading for divergence is visible in the gradient norm hundreds of steps before the loss moves, and stylistic collapse is visible only in the samples."},"body":{"format":"markdown","text":"## Loss is one of six numbers\n\nMost people watch the loss and nothing else. Here is the full set worth having, and what each one tells you that loss does not.\n\n**Gradient norm.** The most under-used signal in fine-tuning. It is the magnitude of the whole gradient vector before clipping, and the shape of its curve is diagnostic:\n\n- A healthy run: moderate at the start, settling to a stable band.\n- Isolated tall spikes: individual pathological examples. Harmless if `max_grad_norm` is clipping them, and worth finding anyway: log the batch index.\n- Steadily climbing: instability. The learning rate is too high and the run is heading for divergence, and you can see it several hundred steps before the loss does.\n- Near zero from the start: nothing is learning. Check trainable parameters before anything else.\n\n**Tokens per optimiser step.** Not logged by default, and it is the true unit of batch size. Compute it from the attention mask and log it once at the start; if it is under a few thousand, your gradients are noisy regardless of what the example count says.\n\n**Token accuracy on unmasked positions.** The fraction of target positions where the argmax is correct. Free to compute, interpretable without exponentiating anything, and less swayed than loss by a handful of very surprising tokens. When loss and token accuracy disagree, a small number of examples are dominating the loss, and finding them usually finds a data problem.\n\n**Learning rate.** Log it, because a schedule that is not doing what you think (no warmup, a decay set for the wrong number of steps) is invisible otherwise and completely obvious on a plot.\n\n**Evaluation loss.** Every hundred steps or so on a held-out slice, to see the training–validation gap open.\n\n**Mean generated length.** Generate on a few fixed prompts at each evaluation and record the mean token count. This is the cheapest detector of the length drift that preference methods cause and that model judges reward.\n\n## The samples\n\nThree fixed prompts, generated at every evaluation, logged as text.\n\nThis is not a metric and it is more informative than all of them. A loss curve cannot show you that the model has started every answer with the same clause, dropped its bullet points, begun repeating itself at the end of long answers, or lost its British spelling. Three samples show all of that at a glance.\n\n> Figure: Six things to log, and what each says that loss does not. Layers, top to bottom: Gradient norm (A run heading for divergence climbs here hundreds of steps before the loss moves. Near zero from the start means nothing is learning.); Tokens per optimiser step (The true unit of batch size, and not logged by default. Under a few thousand, your gradients are noisy whatever the example count says.); Token accuracy on unmasked positions (Free, interpretable without exponentiating anything, and less swayed by a handful of very surprising tokens.); Learning rate (A schedule set for the wrong number of steps is invisible in every other number and obvious on a plot.); Mean generated length (The cheapest detector of the length drift that preference methods cause and that model judges reward.); Three fixed generations (Not a metric, and more informative than all of them. Stylistic collapse appears here and nowhere else.). Pick the three prompts deliberately: one typical input, one edge case, and one that should be declined. Keep the whole log directory beside the checkpoint, because it is a few megabytes and it is the only answer to \"why is version two better than version three\".\n\nPick the three deliberately: one typical input, one edge case, one input that should be declined.\n\n## The tools, all free\n\n**TensorBoard** is the plain answer. It is included with PyTorch, runs entirely locally, needs no account, and handles scalars and text. `report_to=\"tensorboard\"` and `tensorboard --logdir out/`.\n\n**Trackio** is a newer free, open-source tracker with a Weights & Biases-compatible API, runnable locally or hosted on a Hugging Face Space at no cost. Useful when you want a shareable dashboard without a commercial account.\n\n**MLflow** and **Aim** are free and self-hostable, with more experiment-management machinery than a single fine-tune needs but real value once you have twenty runs to compare.\n\n**Weights & Biases** is the commercial standard with a free personal tier. Convenient, and it sends your run metadata to a third party, which for a project training on customer data is a question to answer before the first run rather than after.\n\n## Estimating the run at step 20\n\n```python\n# after 20 steps\nsec_per_step = elapsed / 20\ntotal_hours  = sec_per_step * total_steps / 3600\nprint(f\"{sec_per_step:.2f}s/step → {total_hours:.1f}h → ${total_hours*0.60:.2f}\")\n```\n\nRun this twenty steps in. It tells you whether your six-hour plan is a six-hour plan or a fifty-hour one, while cancelling is still free. Throughput per step is roughly constant after warmup, so the extrapolation is reliable.\n\n## What to keep after the run\n\nThe TensorBoard directory is a few megabytes. Keep it beside the checkpoint, in the same place, forever. Three months later, when somebody asks why version 2 is better than version 3, the answer is in those curves; and if they were only in a terminal you closed, the answer is a shrug."},"check":{"question":"A run's loss is falling smoothly, but the gradient norm has been climbing steadily for 400 steps. What does this most likely mean?","options":[{"text":"The model is learning faster as it adapts, and a rising gradient norm reflects the increasing signal available from the data.","correct":false,"tempting":"Reads a rising norm as progress; gradients typically stabilise or shrink as a model fits, so sustained growth is the opposite signal."},{"text":"Gradient clipping is active, so the reported norm reflects the clipping threshold rather than the true gradient.","correct":false,"tempting":"Confuses the reported pre-clipping norm with the post-clipping value; the logged figure is what clipping is applied to."},{"text":"The evaluation set has drifted from the training distribution, which raises gradient magnitudes on evaluation batches.","correct":false,"tempting":"Evaluation batches produce no gradients at all, so nothing about the eval set can move the training gradient norm."},{"text":"The run is becoming unstable at the current learning rate, and the loss will follow with a spike or divergence later.","correct":true}]},"prev":{"slug":"debugging-a-run","title":"Debugging a run","url":"https://addaly.com/learn/fine-tuning/debugging-a-run"},"next":{"slug":"checkpointing-and-resuming","title":"Checkpointing and resuming","url":"https://addaly.com/learn/fine-tuning/checkpointing-and-resuming"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What to log, and how to read it', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/logging-and-watching (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What to log, and how to read it","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"Gradient norm, tokens per step, token accuracy and three generated samples tell you things loss cannot: a run heading for divergence is visible in the gradient norm hundreds of steps before the loss moves, and stylistic collapse is visible only in the samples.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Six things to log, and what each says that loss does not","layers":[{"label":"Gradient norm","detail":"A run heading for divergence climbs here hundreds of steps before the loss moves. Near zero from the start means nothing is learning."},{"label":"Tokens per optimiser step","detail":"The true unit of batch size, and not logged by default. Under a few thousand, your gradients are noisy whatever the example count says."},{"label":"Token accuracy on unmasked positions","detail":"Free, interpretable without exponentiating anything, and less swayed by a handful of very surprising tokens."},{"label":"Learning rate","detail":"A schedule set for the wrong number of steps is invisible in every other number and obvious on a plot."},{"label":"Mean generated length","detail":"The cheapest detector of the length drift that preference methods cause and that model judges reward."},{"label":"Three fixed generations","detail":"Not a metric, and more informative than all of them. Stylistic collapse appears here and nowhere else."}],"caption":"Pick the three prompts deliberately: one typical input, one edge case, and one that should be declined. Keep the whole log directory beside the checkpoint, because it is a few megabytes and it is the only answer to “why is version two better than version three”."},"alt":"Figure: Six things to log, and what each says that loss does not. Layers, top to bottom: Gradient norm (A run heading for divergence climbs here hundreds of steps before the loss moves. Near zero from the start means nothing is learning.); Tokens per optimiser step (The true unit of batch size, and not logged by default. Under a few thousand, your gradients are noisy whatever the example count says.); Token accuracy on unmasked positions (Free, interpretable without exponentiating anything, and less swayed by a handful of very surprising tokens.); Learning rate (A schedule set for the wrong number of steps is invisible in every other number and obvious on a plot.); Mean generated length (The cheapest detector of the length drift that preference methods cause and that model judges reward.); Three fixed generations (Not a metric, and more informative than all of them. Stylistic collapse appears here and nowhere else.). Pick the three prompts deliberately: one typical input, one edge case, and one that should be declined. Keep the whole log directory beside the checkpoint, because it is a few megabytes and it is the only answer to “why is version two better than version three”."}},"blocks":[{"t":"h2","id":"loss-is-one-of-six-numbers","runs":[{"text":"Loss is one of six numbers"}]},{"t":"p","runs":[{"text":"Most people watch the loss and nothing else. Here is the full set worth having, and what each one tells you that loss does not."}]},{"t":"p","runs":[{"text":"Gradient norm.","b":true},{"text":" The most under-used signal in fine-tuning. It is the magnitude of the whole gradient vector before clipping, and the shape of its curve is diagnostic:"}]},{"t":"ul","items":[[{"text":"A healthy run: moderate at the start, settling to a stable band."}],[{"text":"Isolated tall spikes: individual pathological examples. Harmless if "},{"text":"max_grad_norm","code":true},{"text":" is clipping them, and worth finding anyway: log the batch index."}],[{"text":"Steadily climbing: instability. The learning rate is too high and the run is heading for divergence, and you can see it several hundred steps before the loss does."}],[{"text":"Near zero from the start: nothing is learning. Check trainable parameters before anything else."}]]},{"t":"p","runs":[{"text":"Tokens per optimiser step.","b":true},{"text":" Not logged by default, and it is the true unit of batch size. Compute it from the attention mask and log it once at the start; if it is under a few thousand, your gradients are noisy regardless of what the example count says."}]},{"t":"p","runs":[{"text":"Token accuracy on unmasked positions.","b":true},{"text":" The fraction of target positions where the argmax is correct. Free to compute, interpretable without exponentiating anything, and less swayed than loss by a handful of very surprising tokens. When loss and token accuracy disagree, a small number of examples are dominating the loss, and finding them usually finds a data problem."}]},{"t":"p","runs":[{"text":"Learning rate.","b":true},{"text":" Log it, because a schedule that is not doing what you think (no warmup, a decay set for the wrong number of steps) is invisible otherwise and completely obvious on a plot."}]},{"t":"p","runs":[{"text":"Evaluation loss.","b":true},{"text":" Every hundred steps or so on a held-out slice, to see the training–validation gap open."}]},{"t":"p","runs":[{"text":"Mean generated length.","b":true},{"text":" Generate on a few fixed prompts at each evaluation and record the mean token count. This is the cheapest detector of the length drift that preference methods cause and that model judges reward."}]},{"t":"h2","id":"the-samples","runs":[{"text":"The samples"}]},{"t":"p","runs":[{"text":"Three fixed prompts, generated at every evaluation, logged as text."}]},{"t":"p","runs":[{"text":"This is not a metric and it is more informative than all of them. A loss curve cannot show you that the model has started every answer with the same clause, dropped its bullet points, begun repeating itself at the end of long answers, or lost its British spelling. Three samples show all of that at a glance."}]},{"t":"p","runs":[{"text":"Pick the three deliberately: one typical input, one edge case, one input that should be declined."}]},{"t":"h2","id":"the-tools-all-free","runs":[{"text":"The tools, all free"}]},{"t":"p","runs":[{"text":"TensorBoard","b":true},{"text":" is the plain answer. It is included with PyTorch, runs entirely locally, needs no account, and handles scalars and text. "},{"text":"report_to=\"tensorboard\"","code":true},{"text":" and "},{"text":"tensorboard --logdir out/","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Trackio","b":true},{"text":" is a newer free, open-source tracker with a Weights & Biases-compatible API, runnable locally or hosted on a Hugging Face Space at no cost. Useful when you want a shareable dashboard without a commercial account."}]},{"t":"p","runs":[{"text":"MLflow","b":true},{"text":" and "},{"text":"Aim","b":true},{"text":" are free and self-hostable, with more experiment-management machinery than a single fine-tune needs but real value once you have twenty runs to compare."}]},{"t":"p","runs":[{"text":"Weights & Biases","b":true},{"text":" is the commercial standard with a free personal tier. Convenient, and it sends your run metadata to a third party, which for a project training on customer data is a question to answer before the first run rather than after."}]},{"t":"h2","id":"estimating-the-run-at-step-20","runs":[{"text":"Estimating the run at step 20"}]},{"t":"code","lang":"python","label":"Python","text":"# after 20 steps\nsec_per_step = elapsed / 20\ntotal_hours  = sec_per_step * total_steps / 3600\nprint(f\"{sec_per_step:.2f}s/step → {total_hours:.1f}h → ${total_hours*0.60:.2f}\")"},{"t":"p","runs":[{"text":"Run this twenty steps in. It tells you whether your six-hour plan is a six-hour plan or a fifty-hour one, while cancelling is still free. Throughput per step is roughly constant after warmup, so the extrapolation is reliable."}]},{"t":"h2","id":"what-to-keep-after-the-run","runs":[{"text":"What to keep after the run"}]},{"t":"p","runs":[{"text":"The TensorBoard directory is a few megabytes. Keep it beside the checkpoint, in the same place, forever. Three months later, when somebody asks why version 2 is better than version 3, the answer is in those curves; and if they were only in a terminal you closed, the answer is a shrug."}]}],"check":{"question":[{"text":"A run’s loss is falling smoothly, but the gradient norm has been climbing steadily for 400 steps. What does this most likely mean?"}],"options":[{"text":[{"text":"The model is learning faster as it adapts, and a rising gradient norm reflects the increasing signal available from the data."}],"correct":false,"tempting":[{"text":"Reads a rising norm as progress; gradients typically stabilise or shrink as a model fits, so sustained growth is the opposite signal."}]},{"text":[{"text":"Gradient clipping is active, so the reported norm reflects the clipping threshold rather than the true gradient."}],"correct":false,"tempting":[{"text":"Confuses the reported pre-clipping norm with the post-clipping value; the logged figure is what clipping is applied to."}]},{"text":[{"text":"The evaluation set has drifted from the training distribution, which raises gradient magnitudes on evaluation batches."}],"correct":false,"tempting":[{"text":"Evaluation batches produce no gradients at all, so nothing about the eval set can move the training gradient norm."}]},{"text":[{"text":"The run is becoming unstable at the current learning rate, and the loss will follow with a spike or divergence later."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"debugging-a-run","title":"Debugging a run","minutes":10},"next":{"slug":"checkpointing-and-resuming","title":"Checkpointing and resuming","minutes":8}}}