Gate and up projections

Two independent weight matrices that both expand the residual stream into a wider hidden space — one to pass through SwiGLU unchanged, one to act as its gate.

The feed-forward sub-layer begins with two parallel linear projections, conventionally called gate and up. Both take the same residual-stream vector x of model dimension dmodel and independently project it to the hidden dimension dhidden using separate weight matrices Wgate and Wup: gate = Wgate · x, up = Wup · x. Neither matrix shares weights with the other; they are learned entirely independently and emphasise different features of the input.

The reason for splitting into two rather than one is the SwiGLU activation that follows: one vector supplies the values (up) and the other supplies the gating signal (gate). If they were the same projection the gate could not learn to be independent of the content it is gating — the two roles would be entangled. Keeping them separate allows Wgate to specialise in detecting which dimensions are relevant at each position, while Wup can specialise in representing what those dimensions should actually say.

In the diagram both projections branch upward off the residual backbone simultaneously. They are shown as two lines leaving the same residual-stream node and arriving at the SwiGLU node, where they are combined. The output of each is a vector of d_hidden values — the expanded intermediate representation that only exists inside the feed-forward block.

Related: SwiGLU activation · Down projection · Hidden dimension