Backpropagation

How the gradient of every weight is computed — one backward sweep per step.

Backpropagation is the algorithm that turns a single loss number into a gradient for every weight at once. It is the chain rule applied in reverse: starting from the loss at the output, it walks backward through the network — output projection, then each block from last to first — multiplying local derivatives together to find how much each weight contributed to the error. It reuses the values the forward pass already computed and stored, which is what makes one backward pass about as cheap as one forward pass instead of astronomically expensive.

The direction matters. The forward pass flows input → output (token to logits); backpropagation flows output → input (loss back to the embedding). One forward pass and one backward pass together make one training step, after which the optimizer nudges the weights and the next step begins. Backprop only computes the gradients; it does not change any weight itself — that is the optimization step's job.

Because a hand-written backward pass is easy to get subtly wrong, this project verifies it with a finite-difference gradient check: it perturbs a weight by a tiny amount, measures how the loss actually changes, and confirms that matches the gradient backprop reported. That test must stay green for the training code to be trusted.

In the playground, press Train live and watch the teal backward pulse sweep right→left across the backbone — output to input — once per training step, while the weight chips light up with the gradients it produces.

Related: Gradients · Forward vs backward · Optimization step