Residual add

The skip connection that lets every sub-layer propose an edit rather than a replacement — and lets gradients flow unimpeded to the first layer.

Residual addition is the ⊕ symbol on every transformer block diagram: after a sub-layer computes its output, that output is added to the input vector, not substituted for it. The full update is x ← x + sublayer(norm(x)). The sub-layer only needs to learn the correction — how to adjust what is already there. If the sub-layer has nothing useful to contribute, it can learn to produce near-zero outputs and the stream passes through unchanged. That default-to-identity behaviour is why deep transformers train at all without careful initialisation tricks.

The other critical property is gradient flow. During backpropagation, the gradient of the loss with respect to the residual stream at layer L is the sum of two paths: the gradient that flows back through the sub-layer, plus a direct copy that skips straight over it via the +. That direct copy is always there regardless of how saturated or near-zero the sub-layer's own gradient has become. In a network without residual connections, a sub-layer with small gradients becomes a bottleneck that starves all earlier layers of signal; with residual connections each + is a bypass that keeps the highway open. This is the mechanism that makes 32-layer (or 96-layer) stacks trainable.

Both attention and the feed-forward network sit behind their own residual add in every block. RMSNorm runs before the sub-layer (pre-norm), so the normalisation happens on a fresh copy of the stream before it forks; the raw stream carries forward unnormalised, preserving the magnitude information that accumulated up to that point.

The residual stream enters each block, forks into a normalised path through the sub-layer, and is added back. Because the stream is never replaced — only incremented — the identity path is always open and gradients flow end-to-end.

Related: The residual vector · Logit lens