The loss curve

Reading the live cross-entropy curve as the model trains — down means learning.

The loss curve is a line that traces the model's cross-entropy loss after each training step. Because cross-entropy measures how surprised the model is by the correct next token — higher means more surprised — a falling curve means the model is getting less wrong: it is learning to predict the training text.

The math is direct: at each step the loss is −log p(correcttoken), where p(correcttoken) is the probability the current weights assign to the token that actually came next. A random model over a vocabulary of size V starts near log V nats. As weights improve, the probability concentrates on the right token, its log-probability rises toward zero, and so the loss falls. Real training curves are jagged — each step trains on a different position in the text — but the trend must go down, or something is wrong (learning rate too high, vanishing gradients, a bug in the backward pass).

In the playground the curve updates live on every "Train live" step. It starts high (the model knows nothing), drops steeply in the first few steps as the easiest patterns are learned, then levels off as the remaining error becomes harder to squeeze out. A curve that plateaus high and never falls usually means the corpus is too small, the model is too tiny, or the learning rate needs tuning.

Each step the model sees a token, measures how surprised it was, and nudges its weights — watch the curve fall as those nudges add up.

The loss curve is the primary signal that training is working. It doesn't say which weights changed or by how much — that's what the weight chips and the backward pulse show — but it answers the one question that matters first: is the model getting better?