RMSNorm

Rescaling the vector to unit root-mean-square before each sub-layer (Llama-style pre-norm).

A residual stream that flows through many layers tends to drift in magnitude — each sub-layer adds to it, and without a check the numbers can grow until training becomes unstable. RMSNorm is the cheap, stabilising step that keeps every sub-layer seeing inputs of a consistent scale.

It works in one line. Take the vector x, measure its root-mean-square — rms = √(mean(xᵢ²) + ε) — and divide by it, then multiply by a learned per-dimension gain g:

x_norm = x / rms · g

Unlike the older LayerNorm, it does not subtract the mean or add a bias — it only rescales, never re-centres. That makes it simpler and faster while working just as well in practice, which is why Llama-style models use it.

The playground places an RMSNorm (labelled ‖x‖) twice in every block — once before attention and once before the feed-forward — plus a final one before the logits. This is pre-norm: the vector is normalised on the way into each sub-layer, while the residual stream itself is carried forward unchanged.