Cross-entropy loss

How wrong the predicted next-token distribution is versus the true token — in nats.

Cross-entropy loss is the number the model minimises during training. At each position in the training text there is exactly one correct next token. The model produces a probability distribution over the entire vocabulary — a number between 0 and 1 for every token, summing to 1. Cross-entropy measures how much probability the model placed on the right answer: L = −log p(correct_token). If the model is confident and correct (p near 1), the log is near zero and the loss is tiny. If the model is confident and wrong, or simply unsure (small p), the loss is large.

The formula comes from information theory. −log p is the number of nats needed to encode a surprise of size p — it is the optimal code length for an event that happens with probability p. Cross-entropy is therefore the average code length the model's distribution would need to describe the training data. A perfect model (one that always assigns probability 1 to the true token) has zero cross-entropy. A uniform model over a vocabulary of V tokens has log V nats — that is the ceiling the model starts near, and training's job is to push below it.

Cross-entropy is chosen over other loss functions because language modelling is a classification problem at every step: pick one token from V candidates. Softmax outputs a proper probability distribution and cross-entropy is the natural log-likelihood objective for distributions. The gradient of −log p with respect to the logit of the correct token is simply p − 1, meaning the update always pushes the correct token's logit up (and all others slightly down via the softmax normalisation), scaled by how wrong the current answer is.

In the playground, the number shown next to the curve label is the loss at the current step. Early in training it will sit near log(vocab_size); as training proceeds it falls. When you see it drop by a large amount in a single step, the model just learned something structurally important about the text — a common character pair, a word boundary pattern, or a frequent short phrase.