Gate and up projections

Two independent weight matrices that both expand the residual stream into a wider hidden space — one to pass through SwiGLU unchanged, one to act as its gate.

The feed-forward sub-layer begins with two parallel linear projections, conventionally called gate and up. Both take the same residual-stream vector x of model dimension d_model and independently project it to the hidden dimension d_hidden using separate weight matrices W_gate and W_up: gate = W_gate · x, up = W_up · x. Neither matrix shares weights with the other; they are learned entirely independently and emphasise different features of the input.

The reason for splitting into two rather than one is the SwiGLU activation that follows: one vector supplies the values (up) and the other supplies the gating signal (gate). If they were the same projection the gate could not learn to be independent of the content it is gating — the two roles would be entangled. Keeping them separate allows W_gate to specialise in detecting which dimensions are relevant at each position, while W_up can specialise in representing what those dimensions should actually say.

In the diagram both projections branch upward off the residual backbone simultaneously. They are shown as two lines leaving the same residual-stream node and arriving at the SwiGLU node, where they are combined. The output of each is a vector of d_hidden values — the expanded intermediate representation that only exists inside the feed-forward block.

Related: SwiGLU activation · Down projection · Hidden dimension