The transformer block

The repeatable unit that runs N times: normalization, attention, addition, normalization, feed-forward, addition.

A transformer block is a single layer in the stack of N identical layers. It performs two main operations in sequence, each preceded by normalization and followed by a residual addition (where the output is added back to the input). The pattern is called pre-norm: you normalize first, then apply the operation, then add the result to the original input. This allows gradients to flow cleanly through the deep stack and lets each layer learn only a small correction to its input rather than learning to represent the entire hidden state from scratch.

The first operation is attention: the input is normalized (RMSNorm), then passed through the attention mechanism, and the output is added back to the input. Attention reads all previous tokens (positions 1 through the current position) and computes a weighted blend of their representations — it learns which tokens are relevant to predicting the next one. The second operation is feed-forward: the (now updated) hidden state is normalized again, passed through a small neural network with one hidden layer (commonly a SwiGLU network), and the output is added back. Feed-forward applies the same transformation to every element of the hidden state independently — it has no attention, no position awareness, just learned weights that refine the representation. Together, the two operations are: x = x + Attention(RMSNorm(x)), then x = x + FeedForward(RMSNorm(x)). The two additions ensure that information flows from the block's input directly to its output, making deep networks trainable.

In the diagram, one transformer block appears as a branching unit above the residual stream. The stream enters from the left, splits into the attention branch (upper) and the feed-forward branch (lower), and both outputs reconverge at two addition symbols (⊕) before exiting to the right. The weight matrices (dashed boxes) sit below each branch. When you run generation and the pulse reaches a layer, you see the attention scores light up and the feed-forward weights activate; the output remerges with the residual stream, ready for the next layer.

Related: Layer stack · Forward pass · Parameters