{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"debugging-a-run","title":"Debugging a run","url":"https://addaly.com/learn/fine-tuning/debugging-a-run","position":53,"of":84,"minutes":10,"answer":"Decode the unmasked labels of one real batch before every run (it should print exactly the assistant's answer and nothing else) and if a pipeline cannot memorise eight examples in a hundred steps.","keyIdea":"Decode the unmasked labels of one real batch before every run (it should print exactly the assistant's answer and nothing else) and if a pipeline cannot memorise eight examples in a hundred steps, the problem is the code rather than the data."},"body":{"format":"markdown","text":"## The four checks to run before every job\n\nThese take under a minute together and catch the majority of fine-tuning bugs.\n\n**1. Print one templated example.**\n\n```python\nprint(tok.apply_chat_template(train[0][\"messages\"], tokenize=False))\n```\n\nRead it. Every special token where you expect it, roles marked correctly, exactly one end-of-turn marker at the end.\n\n**2. Decode the unmasked labels of one real batch.**\n\n```python\nb = next(iter(trainer.get_train_dataloader()))\nlbl = b[\"labels\"][0]\nprint(tok.decode([t for t in lbl if t != -100]))\n```\n\nThis should print **exactly the assistant's answer**: not the prompt, not the system message, not nothing. If it prints the whole conversation, your masking is off. If it prints nothing, your response template string does not match. If it prints the answer without a trailing end token, you have the pad-equals-eos problem.\n\nThis single check is the highest-value five seconds in applied fine-tuning.\n\n**3. Print the trainable parameter count.**\n\n```python\nmodel.print_trainable_parameters()\n```\n\nZero means your target modules do not exist in this architecture. A number equal to the full model means your adapter was not applied.\n\n**4. Print the first three token ids.** One BOS, not two.\n\n## The overfit test\n\nBefore a real run, take eight examples and train on them for 100 steps with everything else identical.\n\nThe loss should fall to near zero. The model should reproduce those eight answers nearly verbatim.\n\nIf it cannot memorise eight examples, your pipeline is broken and no amount of data or hyperparameter tuning will help. This test costs two minutes and it separates \"my data is not good enough\" from \"my code does not work\", which are otherwise indistinguishable from the outside and are debugged in completely different ways.\n\n## Symptom to mechanism\n\n**Loss is exactly 0.0.** Every label is -100. The response template did not match; nothing is being trained on.\n\n**Loss is flat and non-zero.** Learning rate far too low, no trainable parameters, or reentrant gradient checkpointing blocking gradients to the adapter. Check the parameter count first.\n\n**Loss drops to near zero within 50 steps.** The answer is visible in the input. Either labels were not shifted, or the completion also appears in the prompt, which happens when a log export includes the response in a context field.\n\n**Loss is NaN.** In fp16, overflow: lower the learning rate, confirm warmup, check for a corrupted example. In bf16, NaN is much rarer and usually means bad data: an empty string, an enormous unicode blob. Find it by logging the index of the batch where it first appears.\n\n**Loss spikes then recovers.** Usually one pathological example. `max_grad_norm=1.0` limits the damage. If it recurs at the same step each epoch, find that example and read it.\n\n**Out of memory at step 40, not step 1.** Your longest sequences are later in the file. Either sort by length to fail fast, or lower `max_length`. With `group_by_length=True` the long batches cluster, which makes this both more likely and more predictable.\n\n**The model never stops generating.** Missing end-of-sequence in training. Cause: pad equals eos and masking by id, or truncation cutting the end token off long examples, or packing without separators.\n\n**Generations are empty.** The model emits end-of-sequence immediately. Usually the reverse masking error: trained on prompts rather than completions.\n\n**Evaluation looks excellent, production does not.** Leakage between your training and evaluation sets, or a formatting difference between how you evaluate and how you serve. Check the second first: print the exact string sent in both paths and diff them.\n\n## Look at generations, not only at loss\n\nEvery two hundred steps, generate on three fixed prompts and log the output.\n\n```python\nclass Samples(TrainerCallback):\n    def on_evaluate(self, args, state, control, model=None, **kw):\n        for p in FIXED_PROMPTS:\n            print(state.global_step, generate(model, p)[:200])\n```\n\nLoss curves hide everything that matters about text. A model that has started repeating itself, dropped its formatting, or begun every answer with the same clause shows all of that in three samples and none of it in the loss.\n\n## The discipline\n\nChange one thing per run. Write down what you changed and what happened, in a file, in the repository. Four runs into a debugging session nobody remembers which combination produced the good result, and the log is what turns a frustrating afternoon into a result you can act on."},"check":{"question":"A run reports a training loss of exactly 0.0 from the very first step. What does this indicate?","options":[{"text":"The model has already learned the task perfectly from pretraining, so there is nothing left to fit.","correct":false,"tempting":"No model achieves exactly zero cross-entropy on real text; even perfect predictions leave a small positive loss."},{"text":"The learning rate is zero, so no weights update and the loss remains at its initial value.","correct":false,"tempting":"A zero learning rate freezes the loss at its initial value, which would be a normal positive number rather than exactly zero."},{"text":"Every label is -100, so nothing contributes to the loss: usually a template that matched nothing.","correct":true},{"text":"The labels were not shifted relative to the inputs, so the model is copying the token it can already see.","correct":false,"tempting":"Names a real bug that produces a near-zero loss after a few steps, but it falls from a normal starting value rather than being zero at step one."}]},"prev":{"slug":"a-config-line-by-line","title":"A working config, line by line","url":"https://addaly.com/learn/fine-tuning/a-config-line-by-line"},"next":{"slug":"logging-and-watching","title":"What to log, and how to read it","url":"https://addaly.com/learn/fine-tuning/logging-and-watching"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Debugging a run', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/debugging-a-run (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Debugging a run","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"Decode the unmasked labels of one real batch before every run (it should print exactly the assistant’s answer and nothing else) and if a pipeline cannot memorise eight examples in a hundred steps, the problem is the code rather than the data.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-four-checks-to-run-before-every-job","runs":[{"text":"The four checks to run before every job"}]},{"t":"p","runs":[{"text":"These take under a minute together and catch the majority of fine-tuning bugs."}]},{"t":"p","runs":[{"text":"1. Print one templated example.","b":true}]},{"t":"code","lang":"python","label":"Python","text":"print(tok.apply_chat_template(train[0][\"messages\"], tokenize=False))"},{"t":"p","runs":[{"text":"Read it. Every special token where you expect it, roles marked correctly, exactly one end-of-turn marker at the end."}]},{"t":"p","runs":[{"text":"2. Decode the unmasked labels of one real batch.","b":true}]},{"t":"code","lang":"python","label":"Python","text":"b = next(iter(trainer.get_train_dataloader()))\nlbl = b[\"labels\"][0]\nprint(tok.decode([t for t in lbl if t != -100]))"},{"t":"p","runs":[{"text":"This should print "},{"text":"exactly the assistant’s answer","b":true},{"text":": not the prompt, not the system message, not nothing. If it prints the whole conversation, your masking is off. If it prints nothing, your response template string does not match. If it prints the answer without a trailing end token, you have the pad-equals-eos problem."}]},{"t":"p","runs":[{"text":"This single check is the highest-value five seconds in applied fine-tuning."}]},{"t":"p","runs":[{"text":"3. Print the trainable parameter count.","b":true}]},{"t":"code","lang":"python","label":"Python","text":"model.print_trainable_parameters()"},{"t":"p","runs":[{"text":"Zero means your target modules do not exist in this architecture. A number equal to the full model means your adapter was not applied."}]},{"t":"p","runs":[{"text":"4. Print the first three token ids.","b":true},{"text":" One BOS, not two."}]},{"t":"h2","id":"the-overfit-test","runs":[{"text":"The overfit test"}]},{"t":"p","runs":[{"text":"Before a real run, take eight examples and train on them for 100 steps with everything else identical."}]},{"t":"p","runs":[{"text":"The loss should fall to near zero. The model should reproduce those eight answers nearly verbatim."}]},{"t":"p","runs":[{"text":"If it cannot memorise eight examples, your pipeline is broken and no amount of data or hyperparameter tuning will help. This test costs two minutes and it separates “my data is not good enough” from “my code does not work”, which are otherwise indistinguishable from the outside and are debugged in completely different ways."}]},{"t":"h2","id":"symptom-to-mechanism","runs":[{"text":"Symptom to mechanism"}]},{"t":"p","runs":[{"text":"Loss is exactly 0.0.","b":true},{"text":" Every label is -100. The response template did not match; nothing is being trained on."}]},{"t":"p","runs":[{"text":"Loss is flat and non-zero.","b":true},{"text":" Learning rate far too low, no trainable parameters, or reentrant gradient checkpointing blocking gradients to the adapter. Check the parameter count first."}]},{"t":"p","runs":[{"text":"Loss drops to near zero within 50 steps.","b":true},{"text":" The answer is visible in the input. Either labels were not shifted, or the completion also appears in the prompt, which happens when a log export includes the response in a context field."}]},{"t":"p","runs":[{"text":"Loss is NaN.","b":true},{"text":" In fp16, overflow: lower the learning rate, confirm warmup, check for a corrupted example. In bf16, NaN is much rarer and usually means bad data: an empty string, an enormous unicode blob. Find it by logging the index of the batch where it first appears."}]},{"t":"p","runs":[{"text":"Loss spikes then recovers.","b":true},{"text":" Usually one pathological example. "},{"text":"max_grad_norm=1.0","code":true},{"text":" limits the damage. If it recurs at the same step each epoch, find that example and read it."}]},{"t":"p","runs":[{"text":"Out of memory at step 40, not step 1.","b":true},{"text":" Your longest sequences are later in the file. Either sort by length to fail fast, or lower "},{"text":"max_length","code":true},{"text":". With "},{"text":"group_by_length=True","code":true},{"text":" the long batches cluster, which makes this both more likely and more predictable."}]},{"t":"p","runs":[{"text":"The model never stops generating.","b":true},{"text":" Missing end-of-sequence in training. Cause: pad equals eos and masking by id, or truncation cutting the end token off long examples, or packing without separators."}]},{"t":"p","runs":[{"text":"Generations are empty.","b":true},{"text":" The model emits end-of-sequence immediately. Usually the reverse masking error: trained on prompts rather than completions."}]},{"t":"p","runs":[{"text":"Evaluation looks excellent, production does not.","b":true},{"text":" Leakage between your training and evaluation sets, or a formatting difference between how you evaluate and how you serve. Check the second first: print the exact string sent in both paths and diff them."}]},{"t":"h2","id":"look-at-generations-not-only-at-loss","runs":[{"text":"Look at generations, not only at loss"}]},{"t":"p","runs":[{"text":"Every two hundred steps, generate on three fixed prompts and log the output."}]},{"t":"code","lang":"python","label":"Python","text":"class Samples(TrainerCallback):\n    def on_evaluate(self, args, state, control, model=None, **kw):\n        for p in FIXED_PROMPTS:\n            print(state.global_step, generate(model, p)[:200])"},{"t":"p","runs":[{"text":"Loss curves hide everything that matters about text. A model that has started repeating itself, dropped its formatting, or begun every answer with the same clause shows all of that in three samples and none of it in the loss."}]},{"t":"h2","id":"the-discipline","runs":[{"text":"The discipline"}]},{"t":"p","runs":[{"text":"Change one thing per run. Write down what you changed and what happened, in a file, in the repository. Four runs into a debugging session nobody remembers which combination produced the good result, and the log is what turns a frustrating afternoon into a result you can act on."}]}],"check":{"question":[{"text":"A run reports a training loss of exactly 0.0 from the very first step. What does this indicate?"}],"options":[{"text":[{"text":"The model has already learned the task perfectly from pretraining, so there is nothing left to fit."}],"correct":false,"tempting":[{"text":"No model achieves exactly zero cross-entropy on real text; even perfect predictions leave a small positive loss."}]},{"text":[{"text":"The learning rate is zero, so no weights update and the loss remains at its initial value."}],"correct":false,"tempting":[{"text":"A zero learning rate freezes the loss at its initial value, which would be a normal positive number rather than exactly zero."}]},{"text":[{"text":"Every label is -100, so nothing contributes to the loss: usually a template that matched nothing."}],"correct":true,"tempting":null},{"text":[{"text":"The labels were not shifted relative to the inputs, so the model is copying the token it can already see."}],"correct":false,"tempting":[{"text":"Names a real bug that produces a near-zero loss after a few steps, but it falls from a normal starting value rather than being zero at step one."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"a-config-line-by-line","title":"A working config, line by line","minutes":10},"next":{"slug":"logging-and-watching","title":"What to log, and how to read it","minutes":8}}}