Hidden dimension

The deliberately wider internal space of the feed-forward sub-layer — why the network expands before it contracts.

The hidden dimension (d_hidden) is the width of the intermediate representation inside the feed-forward block, between the gate/up projections and the down projection. In most transformer configurations d_hidden is set to roughly four times d_model — the GPT-2 standard — or to (8/3) · d_model rounded to a multiple of 64, a common SwiGLU convention that keeps matrix shapes hardware-friendly. Either way, d_hidden > d_model: the block is a bottleneck that deliberately widens then narrows. A teaching-sized model might use a smaller ratio — e.g. d_hidden = 2 × d_model — chosen to keep the parameter count and the diagram readable rather than to match any production convention.

The widening serves a purpose. A linear map from d_model back to d_model can only represent rotations, reflections and scalings — a restricted family of transformations. By projecting up first, the network gains access to a richer set: the SwiGLU non-linearity operates in the wider space, and the down projection then selects a d_model-dimensional slice of what that wider computation produced. Universal approximation arguments formalise this: with enough hidden units, a two-layer network can represent any continuous function; the hidden dimension controls how close the approximation can get.

The practical cost is parameter count. The three feed-forward matrices (W_gate, W_up, W_down) each contain d_model × d_hidden values. At four-times expansion they together account for roughly two-thirds of all parameters in a standard transformer block — more than the attention mechanism. Choosing d_hidden is therefore the main lever for trading model capacity against memory and compute budget.

In the diagram the expansion is visible as the feed-forward branch growing taller than the residual backbone before narrowing back. The wider region is where SwiGLU operates; the backbone width never changes — the hidden dimension is entirely internal to the block.