{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"what-goes-wrong-in-training","title":"Loss is NaN, memory is gone, nothing is learning","url":"https://addaly.com/learn/hugging-face/what-goes-wrong-in-training","position":64,"of":75,"minutes":9,"answer":"Training failures sort into memory, NaN, flat loss, good loss with a useless model, and a training–inference mismatch, and the test that separates a pipeline bug from a data problem is deliberately overfitting ten examples.","keyIdea":"Training failures sort into memory, NaN, flat loss, good loss with a useless model, and a training–inference mismatch, and the test that separates a pipeline bug from a data problem is deliberately overfitting ten examples, because a model that cannot memorise ten rows is not learning at all."},"body":{"format":"markdown","text":"## An ordered list, because guessing is expensive\n\nTraining failures are slower to diagnose than inference failures, because each attempt costs an hour. Work down this list rather than trying things.\n\n## Out of memory\n\nThe most common failure and the most mechanical to fix. In order, because each step costs progressively more:\n\n1. **Reduce batch size, raise gradient accumulation.** Free.\n2. **`gradient_checkpointing=True`.** Costs 20 to 30% speed, saves a lot.\n3. **`optim=\"adamw_bnb_8bit\"`.** Saves about three quarters of the optimiser state.\n4. **Shorten `max_length`.** Check your actual length distribution first: often the 95th percentile is far below what you set.\n5. **LoRA, then QLoRA.** The large wins.\n\nOne detail that catches people: **out of memory during evaluation, not training.** Evaluation batches default to the training batch size and generation builds a KV cache, so the peak can be higher than in training. Set `per_device_eval_batch_size` explicitly, lower.\n\nAnd on a shared or notebook environment, memory from a previous failed run is often still held:\n\n> Figure: The curve that tells you to stop. Line chart of loss against training steps: Training loss (0, 2) (100, 1.55) (200, 1.2) (300, 0.95) (400, 0.78) (500, 0.62) (600, 0.5) (700, 0.41); Validation loss (0, 1.98) (100, 1.6) (200, 1.3) (300, 1.15) (400, 1.14) (500, 1.22) (600, 1.34) (700, 1.47). Training loss keeps falling and validation turns upward at about step 350. Everything after that point is memorisation, and it is visible in the logged numbers long before it is visible in the output. load_best_model_at_end keeps the checkpoint from before the turn, which is not the last one and is usually better.\n\n```python\nimport gc, torch\ndel trainer, model\ngc.collect(); torch.cuda.empty_cache()\n```\n\nIf that does not free it, restart the kernel. Half an hour spent tuning batch size against a leak is half an hour wasted.\n\n## Loss is NaN\n\nThree causes, in order of likelihood:\n\n1. **Learning rate too high.** Drop it tenfold and see.\n2. **fp16 overflow.** Switch to bf16 if the hardware allows. This is the T5-family failure from module three, appearing during training instead of inference.\n3. **Bad data.** An empty string, a row whose labels are all `-100` so the loss is dividing by zero, or a label index outside the model's range. Check:\n\n```python\nbad = [i for i, r in enumerate(ds) if all(l == -100 for l in r[\"labels\"])]\nprint(\"rows with no supervised tokens:\", len(bad))\n```\n\nA row with no unmasked labels is a real and common bug in instruction data: it happens when truncation cuts off the whole assistant turn.\n\n## Loss is completely flat from step one\n\nThis is almost never the learning rate. It is the labels not reaching the loss:\n\n- The column is named `label` rather than `labels`.\n- Every label position is masked to `-100`.\n- With PEFT, no parameters are actually trainable: `print_trainable_parameters()` reporting zero means the config's `target_modules` matched nothing, usually because the architecture's module names differ from the ones you copied.\n- The model was loaded with `torch.no_grad()` somewhere, or inputs do not require gradients under a quantized base, which is what `prepare_model_for_kbit_training` fixes.\n\n## Training loss falls beautifully, the model is useless\n\nThe worst case, and it has three usual causes:\n\n**The model learned the prompt.** Prompt tokens were not masked, so it is being graded on reproducing the questions. Decode the unmasked labels and look.\n\n**Overfitting.** Validation loss will show it. Fewer epochs, more data, lower rank.\n\n**The task was learned trivially.** If every example in one class begins with the same phrase, the model learns the phrase. This is the data-quality lesson arriving late and expensive.\n\n## It works in training and not at inference\n\nAlmost always a mismatch between how examples were assembled for training and how prompts are assembled at inference:\n\n- A different chat template, or none.\n- `add_generation_prompt=True` missing at inference.\n- A system message present during training and absent afterwards.\n- Different truncation, so the model sees a shape it never trained on.\n\nThe test: render one training example and one inference prompt as strings and diff them. They should be identical up to the assistant's turn. This finds the problem in two minutes and people spend days not doing it.\n\n## Catastrophic forgetting\n\nThe model gets better at your task and worse at everything else. Expected when full fine-tuning at a high learning rate. Mitigations, roughly in order: use LoRA rather than full fine-tuning, lower the learning rate, fewer epochs, and mix in some general instruction data alongside your task data.\n\nWorth measuring rather than assuming: keep twenty general prompts unrelated to your task and run them before and after. If the answers have degraded, you have quantified the trade rather than discovered it in production.\n\n## The cheapest habit in this module\n\n**Overfit ten examples deliberately.** Train on ten rows for many epochs and check that the model reproduces them almost exactly. If it cannot memorise ten examples, your pipeline is broken (labels, masking, trainable parameters, something) and no amount of data will help. Ten minutes, and it separates a pipeline bug from a data problem before you spend three hours on the wrong one.\n\n## Do this now\n\nRun the ten-example overfit test on whatever you are about to train. If it works, your plumbing is sound and every later failure is about data or hyperparameters. That is a useful thing to know before hour three."},"check":{"question":"A LoRA fine-tune shows a loss that is flat from the very first step. What should be checked before the learning rate?","options":[{"text":"Whether the learning-rate warmup is too long, so the effective rate is near zero for the logged steps.","correct":false,"tempting":"Warmup would show a rate ramping up in the logs and a loss that begins moving after it."},{"text":"Whether gradient accumulation is set so high that no optimiser step has occurred within the logged window.","correct":false,"tempting":"Plausible arithmetic, but accumulation of that size would be visible in the configuration and still eventually step."},{"text":"Whether the batch size is too small for the loss to be stable enough to show a trend.","correct":false,"tempting":"Small batches make a curve noisy rather than flat."},{"text":"Whether any parameters are trainable, since `target_modules` may have matched nothing.","correct":true}]},"prev":{"slug":"tracking-and-checkpointing","title":"Watching a run you cannot sit next to","url":"https://addaly.com/learn/hugging-face/tracking-and-checkpointing"},"next":{"slug":"publishing-an-adapter","title":"Shipping the thing you trained","url":"https://addaly.com/learn/hugging-face/publishing-an-adapter"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Loss is NaN, memory is gone, nothing is learning', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/what-goes-wrong-in-training (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Loss is NaN, memory is gone, nothing is learning","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"Training failures sort into memory, NaN, flat loss, good loss with a useless model, and a training–inference mismatch, and the test that separates a pipeline bug from a data problem is deliberately overfitting ten examples, because a model that cannot memorise ten rows is not learning at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The curve that tells you to stop","series":[{"label":"Training loss","points":[[0,2],[100,1.55],[200,1.2],[300,0.95],[400,0.78],[500,0.62],[600,0.5],[700,0.41]]},{"label":"Validation loss","points":[[0,1.98],[100,1.6],[200,1.3],[300,1.15],[400,1.14],[500,1.22],[600,1.34],[700,1.47]]}],"xLabel":"training steps","yLabel":"loss","caption":"Training loss keeps falling and validation turns upward at about step 350. Everything after that point is memorisation, and it is visible in the logged numbers long before it is visible in the output. load_best_model_at_end keeps the checkpoint from before the turn, which is not the last one and is usually better."},"alt":"Figure: The curve that tells you to stop. Line chart of loss against training steps: Training loss (0, 2) (100, 1.55) (200, 1.2) (300, 0.95) (400, 0.78) (500, 0.62) (600, 0.5) (700, 0.41); Validation loss (0, 1.98) (100, 1.6) (200, 1.3) (300, 1.15) (400, 1.14) (500, 1.22) (600, 1.34) (700, 1.47). Training loss keeps falling and validation turns upward at about step 350. Everything after that point is memorisation, and it is visible in the logged numbers long before it is visible in the output. load_best_model_at_end keeps the checkpoint from before the turn, which is not the last one and is usually better."}},"blocks":[{"t":"h2","id":"an-ordered-list-because-guessing-is-expensive","runs":[{"text":"An ordered list, because guessing is expensive"}]},{"t":"p","runs":[{"text":"Training failures are slower to diagnose than inference failures, because each attempt costs an hour. Work down this list rather than trying things."}]},{"t":"h2","id":"out-of-memory","runs":[{"text":"Out of memory"}]},{"t":"p","runs":[{"text":"The most common failure and the most mechanical to fix. In order, because each step costs progressively more:"}]},{"t":"ol","start":1,"items":[[{"text":"Reduce batch size, raise gradient accumulation.","b":true},{"text":" Free."}],[{"text":"gradient_checkpointing=True","b":true,"code":true},{"text":".","b":true},{"text":" Costs 20 to 30% speed, saves a lot."}],[{"text":"optim=\"adamw_bnb_8bit\"","b":true,"code":true},{"text":".","b":true},{"text":" Saves about three quarters of the optimiser state."}],[{"text":"Shorten ","b":true},{"text":"max_length","b":true,"code":true},{"text":".","b":true},{"text":" Check your actual length distribution first: often the 95th percentile is far below what you set."}],[{"text":"LoRA, then QLoRA.","b":true},{"text":" The large wins."}]]},{"t":"p","runs":[{"text":"One detail that catches people: "},{"text":"out of memory during evaluation, not training.","b":true},{"text":" Evaluation batches default to the training batch size and generation builds a KV cache, so the peak can be higher than in training. Set "},{"text":"per_device_eval_batch_size","code":true},{"text":" explicitly, lower."}]},{"t":"p","runs":[{"text":"And on a shared or notebook environment, memory from a previous failed run is often still held:"}]},{"t":"code","lang":"python","label":"Python","text":"import gc, torch\ndel trainer, model\ngc.collect(); torch.cuda.empty_cache()"},{"t":"p","runs":[{"text":"If that does not free it, restart the kernel. Half an hour spent tuning batch size against a leak is half an hour wasted."}]},{"t":"h2","id":"loss-is-nan","runs":[{"text":"Loss is NaN"}]},{"t":"p","runs":[{"text":"Three causes, in order of likelihood:"}]},{"t":"ol","start":1,"items":[[{"text":"Learning rate too high.","b":true},{"text":" Drop it tenfold and see."}],[{"text":"fp16 overflow.","b":true},{"text":" Switch to bf16 if the hardware allows. This is the T5-family failure from module three, appearing during training instead of inference."}],[{"text":"Bad data.","b":true},{"text":" An empty string, a row whose labels are all "},{"text":"-100","code":true},{"text":" so the loss is dividing by zero, or a label index outside the model’s range. Check:"}]]},{"t":"code","lang":"python","label":"Python","text":"bad = [i for i, r in enumerate(ds) if all(l == -100 for l in r[\"labels\"])]\nprint(\"rows with no supervised tokens:\", len(bad))"},{"t":"p","runs":[{"text":"A row with no unmasked labels is a real and common bug in instruction data: it happens when truncation cuts off the whole assistant turn."}]},{"t":"h2","id":"loss-is-completely-flat-from-step-one","runs":[{"text":"Loss is completely flat from step one"}]},{"t":"p","runs":[{"text":"This is almost never the learning rate. It is the labels not reaching the loss:"}]},{"t":"ul","items":[[{"text":"The column is named "},{"text":"label","code":true},{"text":" rather than "},{"text":"labels","code":true},{"text":"."}],[{"text":"Every label position is masked to "},{"text":"-100","code":true},{"text":"."}],[{"text":"With PEFT, no parameters are actually trainable: "},{"text":"print_trainable_parameters()","code":true},{"text":" reporting zero means the config’s "},{"text":"target_modules","code":true},{"text":" matched nothing, usually because the architecture’s module names differ from the ones you copied."}],[{"text":"The model was loaded with "},{"text":"torch.no_grad()","code":true},{"text":" somewhere, or inputs do not require gradients under a quantized base, which is what "},{"text":"prepare_model_for_kbit_training","code":true},{"text":" fixes."}]]},{"t":"h2","id":"training-loss-falls-beautifully-the-model-is-useless","runs":[{"text":"Training loss falls beautifully, the model is useless"}]},{"t":"p","runs":[{"text":"The worst case, and it has three usual causes:"}]},{"t":"p","runs":[{"text":"The model learned the prompt.","b":true},{"text":" Prompt tokens were not masked, so it is being graded on reproducing the questions. Decode the unmasked labels and look."}]},{"t":"p","runs":[{"text":"Overfitting.","b":true},{"text":" Validation loss will show it. Fewer epochs, more data, lower rank."}]},{"t":"p","runs":[{"text":"The task was learned trivially.","b":true},{"text":" If every example in one class begins with the same phrase, the model learns the phrase. This is the data-quality lesson arriving late and expensive."}]},{"t":"h2","id":"it-works-in-training-and-not-at-inference","runs":[{"text":"It works in training and not at inference"}]},{"t":"p","runs":[{"text":"Almost always a mismatch between how examples were assembled for training and how prompts are assembled at inference:"}]},{"t":"ul","items":[[{"text":"A different chat template, or none."}],[{"text":"add_generation_prompt=True","code":true},{"text":" missing at inference."}],[{"text":"A system message present during training and absent afterwards."}],[{"text":"Different truncation, so the model sees a shape it never trained on."}]]},{"t":"p","runs":[{"text":"The test: render one training example and one inference prompt as strings and diff them. They should be identical up to the assistant’s turn. This finds the problem in two minutes and people spend days not doing it."}]},{"t":"h2","id":"catastrophic-forgetting","runs":[{"text":"Catastrophic forgetting"}]},{"t":"p","runs":[{"text":"The model gets better at your task and worse at everything else. Expected when full fine-tuning at a high learning rate. Mitigations, roughly in order: use LoRA rather than full fine-tuning, lower the learning rate, fewer epochs, and mix in some general instruction data alongside your task data."}]},{"t":"p","runs":[{"text":"Worth measuring rather than assuming: keep twenty general prompts unrelated to your task and run them before and after. If the answers have degraded, you have quantified the trade rather than discovered it in production."}]},{"t":"h2","id":"the-cheapest-habit-in-this-module","runs":[{"text":"The cheapest habit in this module"}]},{"t":"p","runs":[{"text":"Overfit ten examples deliberately.","b":true},{"text":" Train on ten rows for many epochs and check that the model reproduces them almost exactly. If it cannot memorise ten examples, your pipeline is broken (labels, masking, trainable parameters, something) and no amount of data will help. Ten minutes, and it separates a pipeline bug from a data problem before you spend three hours on the wrong one."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Run the ten-example overfit test on whatever you are about to train. If it works, your plumbing is sound and every later failure is about data or hyperparameters. That is a useful thing to know before hour three."}]}],"check":{"question":[{"text":"A LoRA fine-tune shows a loss that is flat from the very first step. What should be checked before the learning rate?"}],"options":[{"text":[{"text":"Whether the learning-rate warmup is too long, so the effective rate is near zero for the logged steps."}],"correct":false,"tempting":[{"text":"Warmup would show a rate ramping up in the logs and a loss that begins moving after it."}]},{"text":[{"text":"Whether gradient accumulation is set so high that no optimiser step has occurred within the logged window."}],"correct":false,"tempting":[{"text":"Plausible arithmetic, but accumulation of that size would be visible in the configuration and still eventually step."}]},{"text":[{"text":"Whether the batch size is too small for the loss to be stable enough to show a trend."}],"correct":false,"tempting":[{"text":"Small batches make a curve noisy rather than flat."}]},{"text":[{"text":"Whether any parameters are trainable, since "},{"text":"target_modules","code":true},{"text":" may have matched nothing."}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":null},"prev":{"slug":"tracking-and-checkpointing","title":"Watching a run you cannot sit next to","minutes":8},"next":{"slug":"publishing-an-adapter","title":"Shipping the thing you trained","minutes":8}}}