{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"tracking-and-checkpointing","title":"Watching a run you cannot sit next to","url":"https://addaly.com/learn/hugging-face/tracking-and-checkpointing","position":63,"of":75,"minutes":8,"answer":"Log training loss, validation metric, learning rate and gradient norm somewhere that outlives the machine, checkpoint with hub_strategy=\"checkpoint\" so a killed session costs minutes.","keyIdea":"Log training loss, validation metric, learning rate and gradient norm somewhere that outlives the machine, checkpoint with `hub_strategy=\"checkpoint\"` so a killed session costs minutes, and resume with `resume_from_checkpoint` because reloading only the weights restarts the learning-rate schedule."},"body":{"format":"markdown","text":"## What to log, and why the default is not enough\n\nA training run prints a loss every `logging_steps`. That one number tells you whether the run is alive and almost nothing else. Four things are worth having:\n\n1. **Training loss**: is it falling.\n2. **Validation loss or metric**: is it *generalising*, which is a different question.\n3. **Learning rate**: did warmup and decay behave as configured.\n4. **Gradient norm**: spikes here precede a divergence you can otherwise only see after it has happened.\n\nAll four are logged automatically with `report_to` set:\n\n```python\nTrainingArguments(\n    ...,\n    logging_steps=10,\n    eval_strategy=\"steps\", eval_steps=100,\n    report_to=\"tensorboard\",      # or \"trackio\", \"wandb\", \"none\"\n)\n```\n\n## Where to put the logs\n\n**TensorBoard** writes event files into `output_dir`. If `push_to_hub=True` is set, those files go to the model repository and the Hub renders them on a Training metrics tab, so a stranger reading your model card can see the loss curve. Free, no account, no service.\n\n**Trackio** is a lightweight open-source tracker that runs as a Space, storing runs in a dataset repository. It is free, self-hosted and the API mirrors the commercial trackers closely enough that switching is a one-line change.\n\n**Weights & Biases** and similar services are free for personal use and excellent, with the ordinary caveat that your run metadata goes to a third party. On a work project that may need asking about first.\n\nThe important thing is that **the record survives the machine**. A loss curve that existed only in a Colab output cell is gone when the session ends, and with it the ability to compare this run to the next one.\n\n## Reading a loss curve\n\nFour shapes, and what each means:\n\n- **Both losses fall and level off.** Working. Stop when validation stops improving.\n- **Training falls, validation rises.** Overfitting. Fewer epochs, more data, more dropout, or a lower LoRA rank. `load_best_model_at_end` keeps the checkpoint from before the turn.\n- **Both flat.** Learning rate too low, or the labels are not reaching the loss: check the masking. A completely flat loss from step one is nearly always a data bug, not a learning-rate problem.\n- **Loss spikes to NaN.** Learning rate too high, or float16 overflow. Lower the rate; switch to bf16.\n\nA jagged curve is normal at small batch sizes. A curve that is jagged *and* trending flat usually means the effective batch is too small: raise gradient accumulation before you change anything else.\n\n## Checkpoints, and how not to lose a night\n\n```python\nsave_strategy=\"steps\", save_steps=100,\nsave_total_limit=2,\nload_best_model_at_end=True,\npush_to_hub=True, hub_strategy=\"checkpoint\",\n```\n\n`hub_strategy=\"checkpoint\"` pushes the checkpoint to the Hub as it saves, so the run survives the machine dying. `save_total_limit=2` keeps the disk from filling: each checkpoint is a full copy of the trainable state, and on a free instance three of them can be the thing that kills the session.\n\nResuming is one argument:\n\n```python\ntrainer.train(resume_from_checkpoint=True)\n```\n\nThis restores optimiser state and the scheduler position as well as weights. Restarting from a checkpoint by loading only the weights and beginning again resets the learning-rate schedule, which produces a visible jump in the loss and a worse model. Use the argument.\n\n## Seeds and what reproducibility means here\n\n```python\nseed=42, data_seed=42\n```\n\nThose make the run repeatable on the same hardware with the same library versions. Across different GPUs or versions, floating-point accumulation order differs and results drift slightly. That is expected; claiming bit-identical reproducibility across machines is a claim nobody can keep.\n\nWhat you *can* keep: log the versions, the seed, the base model revision and the dataset revision into the run. Then a result is reconstructible even when it is not bit-identical, which is what anybody actually needs.\n\n## Evaluate the model, not the loss\n\nValidation loss going down is necessary and not sufficient. Before declaring a run successful, generate output for twenty held-out inputs and read them. Loss can improve while the model develops a habit you would immediately reject: always answering in English, always producing three bullet points, losing the ability to say it does not know.\n\nThis is twenty minutes and it is the only check that catches that class of problem.\n\n## Do this now\n\nSet `report_to=\"tensorboard\"` and `push_to_hub=True` on your next run, then open the Training metrics tab on the resulting repository. A loss curve attached to the model it produced is a small thing that makes every later comparison possible."},"check":{"question":"A run is interrupted and restarted by loading the last checkpoint's weights and calling `train()` again. The loss jumps upward and the final model is worse than an uninterrupted run. Why?","options":[{"text":"Optimiser state and the learning-rate schedule were not restored, so training restarted with warmup at full rate.","correct":true},{"text":"The checkpoint stored weights in a lower precision, so reloading introduced rounding errors that the loss reflects.","correct":false,"tempting":"Checkpoints save in the training dtype, and rounding would not produce a schedule-shaped jump."},{"text":"Restarting reshuffles the dataset with a new seed, so the model sees examples it has already learned from in a different order.","correct":false,"tempting":"Order changes cause small variance rather than a systematic upward jump and a worse final model."},{"text":"Gradient accumulation counters reset mid-cycle, so the first few updates were computed on partial batches.","correct":false,"tempting":"A real but tiny effect confined to one or two steps."}]},"prev":{"slug":"trl-and-instruction-tuning","title":"TRL, and training on conversations","url":"https://addaly.com/learn/hugging-face/trl-and-instruction-tuning"},"next":{"slug":"what-goes-wrong-in-training","title":"Loss is NaN, memory is gone, nothing is learning","url":"https://addaly.com/learn/hugging-face/what-goes-wrong-in-training"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Watching a run you cannot sit next to', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/tracking-and-checkpointing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Watching a run you cannot sit next to","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"Log training loss, validation metric, learning rate and gradient norm somewhere that outlives the machine, checkpoint with `hub_strategy=\"checkpoint\"` so a killed session costs minutes, and resume with `resume_from_checkpoint` because reloading only the weights restarts the learning-rate schedule.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-to-log-and-why-the-default-is-not-enough","runs":[{"text":"What to log, and why the default is not enough"}]},{"t":"p","runs":[{"text":"A training run prints a loss every "},{"text":"logging_steps","code":true},{"text":". That one number tells you whether the run is alive and almost nothing else. Four things are worth having:"}]},{"t":"ol","start":1,"items":[[{"text":"Training loss","b":true},{"text":": is it falling."}],[{"text":"Validation loss or metric","b":true},{"text":": is it "},{"text":"generalising","i":true},{"text":", which is a different question."}],[{"text":"Learning rate","b":true},{"text":": did warmup and decay behave as configured."}],[{"text":"Gradient norm","b":true},{"text":": spikes here precede a divergence you can otherwise only see after it has happened."}]]},{"t":"p","runs":[{"text":"All four are logged automatically with "},{"text":"report_to","code":true},{"text":" set:"}]},{"t":"code","lang":"python","label":"Python","text":"TrainingArguments(\n    ...,\n    logging_steps=10,\n    eval_strategy=\"steps\", eval_steps=100,\n    report_to=\"tensorboard\",      # or \"trackio\", \"wandb\", \"none\"\n)"},{"t":"h2","id":"where-to-put-the-logs","runs":[{"text":"Where to put the logs"}]},{"t":"p","runs":[{"text":"TensorBoard","b":true},{"text":" writes event files into "},{"text":"output_dir","code":true},{"text":". If "},{"text":"push_to_hub=True","code":true},{"text":" is set, those files go to the model repository and the Hub renders them on a Training metrics tab, so a stranger reading your model card can see the loss curve. Free, no account, no service."}]},{"t":"p","runs":[{"text":"Trackio","b":true},{"text":" is a lightweight open-source tracker that runs as a Space, storing runs in a dataset repository. It is free, self-hosted and the API mirrors the commercial trackers closely enough that switching is a one-line change."}]},{"t":"p","runs":[{"text":"Weights & Biases","b":true},{"text":" and similar services are free for personal use and excellent, with the ordinary caveat that your run metadata goes to a third party. On a work project that may need asking about first."}]},{"t":"p","runs":[{"text":"The important thing is that "},{"text":"the record survives the machine","b":true},{"text":". A loss curve that existed only in a Colab output cell is gone when the session ends, and with it the ability to compare this run to the next one."}]},{"t":"h2","id":"reading-a-loss-curve","runs":[{"text":"Reading a loss curve"}]},{"t":"p","runs":[{"text":"Four shapes, and what each means:"}]},{"t":"ul","items":[[{"text":"Both losses fall and level off.","b":true},{"text":" Working. Stop when validation stops improving."}],[{"text":"Training falls, validation rises.","b":true},{"text":" Overfitting. Fewer epochs, more data, more dropout, or a lower LoRA rank. "},{"text":"load_best_model_at_end","code":true},{"text":" keeps the checkpoint from before the turn."}],[{"text":"Both flat.","b":true},{"text":" Learning rate too low, or the labels are not reaching the loss: check the masking. A completely flat loss from step one is nearly always a data bug, not a learning-rate problem."}],[{"text":"Loss spikes to NaN.","b":true},{"text":" Learning rate too high, or float16 overflow. Lower the rate; switch to bf16."}]]},{"t":"p","runs":[{"text":"A jagged curve is normal at small batch sizes. A curve that is jagged "},{"text":"and","i":true},{"text":" trending flat usually means the effective batch is too small: raise gradient accumulation before you change anything else."}]},{"t":"h2","id":"checkpoints-and-how-not-to-lose-a-night","runs":[{"text":"Checkpoints, and how not to lose a night"}]},{"t":"code","lang":"python","label":"Python","text":"save_strategy=\"steps\", save_steps=100,\nsave_total_limit=2,\nload_best_model_at_end=True,\npush_to_hub=True, hub_strategy=\"checkpoint\","},{"t":"p","runs":[{"text":"hub_strategy=\"checkpoint\"","code":true},{"text":" pushes the checkpoint to the Hub as it saves, so the run survives the machine dying. "},{"text":"save_total_limit=2","code":true},{"text":" keeps the disk from filling: each checkpoint is a full copy of the trainable state, and on a free instance three of them can be the thing that kills the session."}]},{"t":"p","runs":[{"text":"Resuming is one argument:"}]},{"t":"code","lang":"python","label":"Python","text":"trainer.train(resume_from_checkpoint=True)"},{"t":"p","runs":[{"text":"This restores optimiser state and the scheduler position as well as weights. Restarting from a checkpoint by loading only the weights and beginning again resets the learning-rate schedule, which produces a visible jump in the loss and a worse model. Use the argument."}]},{"t":"h2","id":"seeds-and-what-reproducibility-means-here","runs":[{"text":"Seeds and what reproducibility means here"}]},{"t":"code","lang":"python","label":"Python","text":"seed=42, data_seed=42"},{"t":"p","runs":[{"text":"Those make the run repeatable on the same hardware with the same library versions. Across different GPUs or versions, floating-point accumulation order differs and results drift slightly. That is expected; claiming bit-identical reproducibility across machines is a claim nobody can keep."}]},{"t":"p","runs":[{"text":"What you "},{"text":"can","i":true},{"text":" keep: log the versions, the seed, the base model revision and the dataset revision into the run. Then a result is reconstructible even when it is not bit-identical, which is what anybody actually needs."}]},{"t":"h2","id":"evaluate-the-model-not-the-loss","runs":[{"text":"Evaluate the model, not the loss"}]},{"t":"p","runs":[{"text":"Validation loss going down is necessary and not sufficient. Before declaring a run successful, generate output for twenty held-out inputs and read them. Loss can improve while the model develops a habit you would immediately reject: always answering in English, always producing three bullet points, losing the ability to say it does not know."}]},{"t":"p","runs":[{"text":"This is twenty minutes and it is the only check that catches that class of problem."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Set "},{"text":"report_to=\"tensorboard\"","code":true},{"text":" and "},{"text":"push_to_hub=True","code":true},{"text":" on your next run, then open the Training metrics tab on the resulting repository. A loss curve attached to the model it produced is a small thing that makes every later comparison possible."}]}],"check":{"question":[{"text":"A run is interrupted and restarted by loading the last checkpoint’s weights and calling "},{"text":"train()","code":true},{"text":" again. The loss jumps upward and the final model is worse than an uninterrupted run. Why?"}],"options":[{"text":[{"text":"Optimiser state and the learning-rate schedule were not restored, so training restarted with warmup at full rate."}],"correct":true,"tempting":null},{"text":[{"text":"The checkpoint stored weights in a lower precision, so reloading introduced rounding errors that the loss reflects."}],"correct":false,"tempting":[{"text":"Checkpoints save in the training dtype, and rounding would not produce a schedule-shaped jump."}]},{"text":[{"text":"Restarting reshuffles the dataset with a new seed, so the model sees examples it has already learned from in a different order."}],"correct":false,"tempting":[{"text":"Order changes cause small variance rather than a systematic upward jump and a worse final model."}]},{"text":[{"text":"Gradient accumulation counters reset mid-cycle, so the first few updates were computed on partial batches."}],"correct":false,"tempting":[{"text":"A real but tiny effect confined to one or two steps."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"trl-and-instruction-tuning","title":"TRL, and training on conversations","minutes":9},"next":{"slug":"what-goes-wrong-in-training","title":"Loss is NaN, memory is gone, nothing is learning","minutes":9}}}