Reading the loss number

What 1.8 means

Cross-entropy loss is the average negative log-probability the model assigned to the tokens that actually came next. That sentence is invertible, which makes the number readable.

A loss of 1.8 means exp(-1.8) ≈ 0.165: on average, the model gave the correct next token about a 16.5% chance. A loss of 0.9 means about 41%. A loss of 0.3 means about 74%.

Perplexity is just the same number exponentiated the other way: exp(1.8) ≈ 6.0, read as “the model is about as uncertain as if it were choosing uniformly among six options at each step”. Perplexity and loss carry identical information; perplexity is easier to say out loud and easier to misuse.

Why your loss cannot be compared to anyone else’s

This is the part that causes the most confusion on forums, and it has four separate causes.

Different tokenizers. Loss is per token. A tokenizer that splits Hindi text into three times as many tokens as another will produce a different loss on the same sentences, because it is being asked an easier question more often. Cross-model loss comparisons are meaningless unless the tokenizers match. Bits per byte is the fix if you need to compare, and almost nobody does.

Different masking. A run that computes loss over prompt and completion is averaging in a lot of easy tokens: the system prompt is identical in every example, so the model predicts it almost perfectly after a few dozen steps. That drags the average down. Switch on completion-only masking and your loss will jump, and nothing has got worse.

Different data. Loss on your customer support transcripts and loss on a coding dataset are not on a common scale. Repetitive, templated text has low loss because it is predictable.

Different packing. Packed sequences that let attention cross document boundaries make the model predict the first tokens of document B having read document A, which is both harmful and loss-affecting.

So: your loss curve is comparable to your own loss curve, on the same data with the same masking and tokenizer. That is all. A number like “we got down to 0.42” tells a stranger nothing.

Loss is not your objective

The deeper point, and the one that decides projects. You are minimising cross-entropy because it is differentiable and cheap, not because it is what you want. What you want is for the model to route tickets correctly, or write a summary a person accepts. Those are not differentiable, so you optimise a proxy and check the real thing separately.

The proxy and the goal come apart in both directions:

• Loss improves, the task does not. The model learns your formatting habits, your punctuation, the boilerplate at the top of every answer. All highly predictable, all worth a lot of loss, all irrelevant.
• Loss worsens, the task improves. This one surprises people. Validation loss can rise while accuracy on your actual metric keeps climbing, because the model becomes more confident: sharper distributions mean bigger penalties on the items it still gets wrong, even as it gets more items right. If you early-stop on validation loss you will throw away the better model.

The rule: early-stop and select checkpoints on a task metric, not on loss. Loss is for detecting that something is broken, not for deciding what to ship.

The shapes that mean something

Some loss-curve readings are diagnostic:

• Flat from step zero. The learning rate is too small, the mask is removing everything, or nothing is actually trainable. Print the count of trainable parameters; on a misconfigured LoRA it is sometimes zero.
• A vertical drop to near zero in the first hundred steps. The answer is visible in the input. Usually the labels were not shifted, or the completion is present in the prompt.
• NaN. Almost always fp16 overflow, a corrupted example, or a learning rate an order of magnitude too high.
• A sawtooth with a period of exactly one epoch. The model is memorising; each epoch it recognises examples it has seen.

One number to log alongside it

Log token-level accuracy: the fraction of masked positions where the argmax token is the correct one. It is free to compute, it is on a scale you can interpret without exponentiating anything, and it is far less sensitive to a handful of very surprising tokens dominating the average. When loss and token accuracy disagree, the disagreement is itself informative: loss dominated by a few outliers is often a data problem hiding in one or two examples.