{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-loss-you-are-minimising","title":"Reading the loss number","url":"https://addaly.com/learn/fine-tuning/the-loss-you-are-minimising","position":11,"of":84,"minutes":9,"answer":"Cross-entropy is readable as a probability (exp(-loss) is the average chance given to the right token) but it is comparable only against your own runs.","keyIdea":"Cross-entropy is readable as a probability (exp(-loss) is the average chance given to the right token) but it is comparable only against your own runs, and because it is a proxy you must select checkpoints on a task metric, since validation loss can rise while the metric you care about improves."},"body":{"format":"markdown","text":"## What 1.8 means\n\nCross-entropy loss is the average negative log-probability the model assigned to the tokens that actually came next. That sentence is invertible, which makes the number readable.\n\nA loss of 1.8 means `exp(-1.8) ≈ 0.165`: on average, the model gave the correct next token about a 16.5% chance. A loss of 0.9 means about 41%. A loss of 0.3 means about 74%.\n\nPerplexity is just the same number exponentiated the other way: `exp(1.8) ≈ 6.0`, read as \"the model is about as uncertain as if it were choosing uniformly among six options at each step\". Perplexity and loss carry identical information; perplexity is easier to say out loud and easier to misuse.\n\n## Why your loss cannot be compared to anyone else's\n\nThis is the part that causes the most confusion on forums, and it has four separate causes.\n\n**Different tokenizers.** Loss is per token. A tokenizer that splits Hindi text into three times as many tokens as another will produce a different loss on the same sentences, because it is being asked an easier question more often. Cross-model loss comparisons are meaningless unless the tokenizers match. Bits per byte is the fix if you need to compare, and almost nobody does.\n\n**Different masking.** A run that computes loss over prompt *and* completion is averaging in a lot of easy tokens: the system prompt is identical in every example, so the model predicts it almost perfectly after a few dozen steps. That drags the average down. Switch on completion-only masking and your loss will *jump*, and nothing has got worse.\n\n**Different data.** Loss on your customer support transcripts and loss on a coding dataset are not on a common scale. Repetitive, templated text has low loss because it is predictable.\n\n**Different packing.** Packed sequences that let attention cross document boundaries make the model predict the first tokens of document B having read document A, which is both harmful and loss-affecting.\n\nSo: your loss curve is comparable to *your own* loss curve, on the same data with the same masking and tokenizer. That is all. A number like \"we got down to 0.42\" tells a stranger nothing.\n\n## Loss is not your objective\n\nThe deeper point, and the one that decides projects. You are minimising cross-entropy because it is differentiable and cheap, not because it is what you want. What you want is for the model to route tickets correctly, or write a summary a person accepts. Those are not differentiable, so you optimise a proxy and check the real thing separately.\n\nThe proxy and the goal come apart in both directions:\n\n- **Loss improves, the task does not.** The model learns your formatting habits, your punctuation, the boilerplate at the top of every answer. All highly predictable, all worth a lot of loss, all irrelevant.\n- **Loss worsens, the task improves.** This one surprises people. Validation loss can rise while accuracy on your actual metric keeps climbing, because the model becomes more confident: sharper distributions mean bigger penalties on the items it still gets wrong, even as it gets more items right. If you early-stop on validation loss you will throw away the better model.\n\nThe rule: early-stop and select checkpoints on a task metric, not on loss. Loss is for detecting that something is broken, not for deciding what to ship.\n\n## The shapes that mean something\n\nSome loss-curve readings are diagnostic:\n\n- **Flat from step zero.** The learning rate is too small, the mask is removing everything, or nothing is actually trainable. Print the count of trainable parameters; on a misconfigured LoRA it is sometimes zero.\n- **A vertical drop to near zero in the first hundred steps.** The answer is visible in the input. Usually the labels were not shifted, or the completion is present in the prompt.\n- **NaN.** Almost always fp16 overflow, a corrupted example, or a learning rate an order of magnitude too high.\n- **A sawtooth with a period of exactly one epoch.** The model is memorising; each epoch it recognises examples it has seen.\n\n## One number to log alongside it\n\nLog token-level accuracy: the fraction of masked positions where the argmax token is the correct one. It is free to compute, it is on a scale you can interpret without exponentiating anything, and it is far less sensitive to a handful of very surprising tokens dominating the average. When loss and token accuracy disagree, the disagreement is itself informative: loss dominated by a few outliers is often a data problem hiding in one or two examples."},"check":{"question":"A team switches on completion-only loss masking and their training loss immediately jumps from 0.6 to 1.9. What has happened?","options":[{"text":"The easy, predictable prompt tokens were holding the average down; removing them leaves the harder ones.","correct":true},{"text":"Masked positions still contribute a small penalty, so removing them from the average leaves a higher residual loss.","correct":false,"tempting":"Misunderstands ignore_index, which removes positions from the computation entirely rather than down-weighting them."},{"text":"The masking is misconfigured and is now scoring the model on the prompt tokens instead of the completion.","correct":false,"tempting":"Reads the jump as a bug; the same direction of change is exactly what a correct switch produces, so the number alone cannot support this conclusion."},{"text":"Gradient magnitudes rise when fewer positions contribute, which inflates the reported loss through the optimiser's normalisation.","correct":false,"tempting":"Confuses the reported loss with gradient scale; loss is computed before any optimiser step and is unaffected by it."}]},"prev":{"slug":"one-training-step","title":"One training step, taken apart","url":"https://addaly.com/learn/fine-tuning/one-training-step"},"next":{"slug":"optimisers-and-their-memory","title":"Optimisers, and what their state costs","url":"https://addaly.com/learn/fine-tuning/optimisers-and-their-memory"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reading the loss number', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-loss-you-are-minimising (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reading the loss number","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"Cross-entropy is readable as a probability (exp(-loss) is the average chance given to the right token) but it is comparable only against your own runs, and because it is a proxy you must select checkpoints on a task metric, since validation loss can rise while the metric you care about improves.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-1-8-means","runs":[{"text":"What 1.8 means"}]},{"t":"p","runs":[{"text":"Cross-entropy loss is the average negative log-probability the model assigned to the tokens that actually came next. That sentence is invertible, which makes the number readable."}]},{"t":"p","runs":[{"text":"A loss of 1.8 means "},{"text":"exp(-1.8) ≈ 0.165","code":true},{"text":": on average, the model gave the correct next token about a 16.5% chance. A loss of 0.9 means about 41%. A loss of 0.3 means about 74%."}]},{"t":"p","runs":[{"text":"Perplexity is just the same number exponentiated the other way: "},{"text":"exp(1.8) ≈ 6.0","code":true},{"text":", read as “the model is about as uncertain as if it were choosing uniformly among six options at each step”. Perplexity and loss carry identical information; perplexity is easier to say out loud and easier to misuse."}]},{"t":"h2","id":"why-your-loss-cannot-be-compared-to-anyone-else-s","runs":[{"text":"Why your loss cannot be compared to anyone else’s"}]},{"t":"p","runs":[{"text":"This is the part that causes the most confusion on forums, and it has four separate causes."}]},{"t":"p","runs":[{"text":"Different tokenizers.","b":true},{"text":" Loss is per token. A tokenizer that splits Hindi text into three times as many tokens as another will produce a different loss on the same sentences, because it is being asked an easier question more often. Cross-model loss comparisons are meaningless unless the tokenizers match. Bits per byte is the fix if you need to compare, and almost nobody does."}]},{"t":"p","runs":[{"text":"Different masking.","b":true},{"text":" A run that computes loss over prompt "},{"text":"and","i":true},{"text":" completion is averaging in a lot of easy tokens: the system prompt is identical in every example, so the model predicts it almost perfectly after a few dozen steps. That drags the average down. Switch on completion-only masking and your loss will "},{"text":"jump","i":true},{"text":", and nothing has got worse."}]},{"t":"p","runs":[{"text":"Different data.","b":true},{"text":" Loss on your customer support transcripts and loss on a coding dataset are not on a common scale. Repetitive, templated text has low loss because it is predictable."}]},{"t":"p","runs":[{"text":"Different packing.","b":true},{"text":" Packed sequences that let attention cross document boundaries make the model predict the first tokens of document B having read document A, which is both harmful and loss-affecting."}]},{"t":"p","runs":[{"text":"So: your loss curve is comparable to "},{"text":"your own","i":true},{"text":" loss curve, on the same data with the same masking and tokenizer. That is all. A number like “we got down to 0.42” tells a stranger nothing."}]},{"t":"h2","id":"loss-is-not-your-objective","runs":[{"text":"Loss is not your objective"}]},{"t":"p","runs":[{"text":"The deeper point, and the one that decides projects. You are minimising cross-entropy because it is differentiable and cheap, not because it is what you want. What you want is for the model to route tickets correctly, or write a summary a person accepts. Those are not differentiable, so you optimise a proxy and check the real thing separately."}]},{"t":"p","runs":[{"text":"The proxy and the goal come apart in both directions:"}]},{"t":"ul","items":[[{"text":"Loss improves, the task does not.","b":true},{"text":" The model learns your formatting habits, your punctuation, the boilerplate at the top of every answer. All highly predictable, all worth a lot of loss, all irrelevant."}],[{"text":"Loss worsens, the task improves.","b":true},{"text":" This one surprises people. Validation loss can rise while accuracy on your actual metric keeps climbing, because the model becomes more confident: sharper distributions mean bigger penalties on the items it still gets wrong, even as it gets more items right. If you early-stop on validation loss you will throw away the better model."}]]},{"t":"p","runs":[{"text":"The rule: early-stop and select checkpoints on a task metric, not on loss. Loss is for detecting that something is broken, not for deciding what to ship."}]},{"t":"h2","id":"the-shapes-that-mean-something","runs":[{"text":"The shapes that mean something"}]},{"t":"p","runs":[{"text":"Some loss-curve readings are diagnostic:"}]},{"t":"ul","items":[[{"text":"Flat from step zero.","b":true},{"text":" The learning rate is too small, the mask is removing everything, or nothing is actually trainable. Print the count of trainable parameters; on a misconfigured LoRA it is sometimes zero."}],[{"text":"A vertical drop to near zero in the first hundred steps.","b":true},{"text":" The answer is visible in the input. Usually the labels were not shifted, or the completion is present in the prompt."}],[{"text":"NaN.","b":true},{"text":" Almost always fp16 overflow, a corrupted example, or a learning rate an order of magnitude too high."}],[{"text":"A sawtooth with a period of exactly one epoch.","b":true},{"text":" The model is memorising; each epoch it recognises examples it has seen."}]]},{"t":"h2","id":"one-number-to-log-alongside-it","runs":[{"text":"One number to log alongside it"}]},{"t":"p","runs":[{"text":"Log token-level accuracy: the fraction of masked positions where the argmax token is the correct one. It is free to compute, it is on a scale you can interpret without exponentiating anything, and it is far less sensitive to a handful of very surprising tokens dominating the average. When loss and token accuracy disagree, the disagreement is itself informative: loss dominated by a few outliers is often a data problem hiding in one or two examples."}]}],"check":{"question":[{"text":"A team switches on completion-only loss masking and their training loss immediately jumps from 0.6 to 1.9. What has happened?"}],"options":[{"text":[{"text":"The easy, predictable prompt tokens were holding the average down; removing them leaves the harder ones."}],"correct":true,"tempting":null},{"text":[{"text":"Masked positions still contribute a small penalty, so removing them from the average leaves a higher residual loss."}],"correct":false,"tempting":[{"text":"Misunderstands ignore_index, which removes positions from the computation entirely rather than down-weighting them."}]},{"text":[{"text":"The masking is misconfigured and is now scoring the model on the prompt tokens instead of the completion."}],"correct":false,"tempting":[{"text":"Reads the jump as a bug; the same direction of change is exactly what a correct switch produces, so the number alone cannot support this conclusion."}]},{"text":[{"text":"Gradient magnitudes rise when fewer positions contribute, which inflates the reported loss through the optimiser’s normalisation."}],"correct":false,"tempting":[{"text":"Confuses the reported loss with gradient scale; loss is computed before any optimiser step and is unaffected by it."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"one-training-step","title":"One training step, taken apart","minutes":10},"next":{"slug":"optimisers-and-their-memory","title":"Optimisers, and what their state costs","minutes":9}}}