Forward vs backward

The forward pass computes the model's output; the backward pass computes how to improve every weight.

The forward pass is the normal inference sweep: tokens flow left to right through embeddings, each transformer block (attention then feed-forward), a final normalisation, and a projection onto vocabulary logits. Every intermediate tensor — the residual stream at each layer, the query/key/value matrices, the attention weights, the feed-forward activations — is computed and (during training) kept in memory. The forward pass ends with a probability distribution over the next token and a cross-entropy loss number.

The backward pass is the mirror sweep that runs only during training. Starting from the scalar loss, it propagates gradients right to left through every operation the forward pass performed, using the chain rule of calculus: ∂L/∂x = (∂L/∂y) × (∂y/∂x). Each layer receives the gradient of the loss with respect to its output, multiplies it by its own local derivative, and passes the result back to the layer before it. The matrices that need updating — the weight matrices in every attention head and feed-forward layer — accumulate gradients as the backward pass crosses them. At the end, every weight has a ∂L/∂w ready for the optimizer step.

This is why the forward pass must retain its intermediate tensors: the backward pass needs them to compute local derivatives. A feed-forward layer's gradient depends on its activation output; an attention layer's gradient depends on the softmax weights it computed. Training therefore uses roughly twice the memory of inference.

In the playground the backward pass is visible as a pulse that sweeps the diagram right to left after the forward arrow finishes. The weight chips (dashed outlines below the backbone) highlight when the backward pass crosses their layer — brighter means a larger gradient magnitude, so that weight is being updated more aggressively this step. The full weight update only lands after the pulse completes and the optimizer applies the step.