Gradients

For each weight, the direction and amount to nudge it to lower the loss.

A gradient is one number per weight: the partial derivative of the loss with respect to that weight, ∂loss/∂w. Its sign says which way to move the weight to reduce the loss, and its magnitude says how strongly that weight affected the loss on this batch. A weight with a large gradient is being told "you mattered a lot here, change more"; a weight with a near-zero gradient is left almost alone. The optimization step then walks every weight a little way down its gradient.

Gradients are not all equal. Early in training the output projection and the layers nearest the loss usually see the loudest gradients, because they sit closest to where the error is measured; deeper-in weights get smaller, noisier signals. Watching which weights light up tells you where the model is currently learning. In this tiny model — only on the order of a hundred thousand weights across 3 layers — you can actually see this happen per step rather than as an aggregate.

In the playground, press Train live. Each weight chip (the dashed boxes below the residual backbone) glows by its gradient magnitude, normalised so the loudest weight this step is brightest and the quiet ones stay dim. The status line names the single weight currently receiving the largest gradient — the largest ∇ callout — alongside the loss. So you read, step by step, both how wrong the model is and which weight is being corrected hardest.

Related: Backpropagation · Optimization step · The loss curve