Layer stack

Repeating the transformer block N times, with each layer building on the output of the previous one.

Stacking identical transformer blocks is what makes a deep transformer powerful. Even a small stack — a handful of layers, say 3 — already shows the idea, and each layer receives the output of the layer before it. Layer 0 reads the embedded input token and produces a refined hidden state. Layer 1 reads that hidden state (and applies attention over all previous tokens) and produces a further refinement. By the last layer the model has had one pass per layer to read and transform the input, accumulating information and structure. Early layers often learn simple patterns (e.g., which character tends to follow which), while later layers compose those into longer-range structure. The stack is the main source of the model's expressiveness: a single layer can do only what it learns; several layers can compose those learnings, each specialising and delegating to the next. Production LLMs push the same idea to dozens or hundreds of layers.

Depth comes at a cost in parameters and computation, but provides rewards in capability. A 1-layer transformer cannot reason as deeply as a many-layer one, no matter how wide (how many parameters per layer). The residual additions between layers are what make deep stacks trainable: without them, gradients would vanish as they backpropagate through the matrix multiplications of every layer. With them, gradients can "shortcut" directly from the output back to any layer, so even very deep networks learn well. The stack is the heart of the transformer's depth.

In the diagram, the stack appears as a sequence of transformer blocks, left-to-right. The input token enters on the left, flows through Block 0, then Block 1, and so on, exiting Block N-1 on the right. The pulse marks which block is currently active as you generate. Residual streams (the horizontal lines) connect each block to the next; the hidden state evolves as it moves right. You can pause generation, zoom in on any layer, and see the actual activations flowing through — the intermediate vectors the model computed at that stage.

Related: Transformer block · Forward pass · The residual stream