Hidden dimension

The deliberately wider internal space of the feed-forward sub-layer — why the network expands before it contracts.

The hidden dimension (dhidden) is the width of the intermediate representation inside the feed-forward block, between the gate/up projections and the down projection. In most transformer configurations dhidden is set to roughly four times dmodel — the GPT-2 standard — or to (8/3) · dmodel rounded to a multiple of 64, a common SwiGLU convention that keeps matrix shapes hardware-friendly. Either way, dhidden > dmodel: the block is a bottleneck that deliberately widens then narrows. A teaching-sized model might use a smaller ratio — e.g. dhidden = 2 × dmodel — chosen to keep the parameter count and the diagram readable rather than to match any production convention.

The widening serves a purpose. A linear map from dmodel back to dmodel can only represent rotations, reflections and scalings — a restricted family of transformations. By projecting up first, the network gains access to a richer set: the SwiGLU non-linearity operates in the wider space, and the down projection then selects a d_model-dimensional slice of what that wider computation produced. Universal approximation arguments formalise this: with enough hidden units, a two-layer network can represent any continuous function; the hidden dimension controls how close the approximation can get.

The practical cost is parameter count. The three feed-forward matrices (Wgate, Wup, Wdown) each contain dmodel × dhidden values. At four-times expansion they together account for roughly two-thirds of all parameters in a standard transformer block — more than the attention mechanism. Choosing dhidden is therefore the main lever for trading model capacity against memory and compute budget.

In the diagram the expansion is visible as the feed-forward branch growing taller than the residual backbone before narrowing back. The wider region is where SwiGLU operates; the backbone width never changes — the hidden dimension is entirely internal to the block.