Down projection

A single linear map that compresses the wide hidden-dimension vector back to model dimension and deposits the result into the residual stream.

The down projection is the final step of the feed-forward sub-layer. After the gate and up projections have expanded the residual stream to the hidden dimension and SwiGLU has selectively combined them, a single weight matrix Wdown maps the resulting dhidden vector back to dmodel: output = Wdown · swiglu_out. This is a pure linear projection — no activation, no gating — and the result is added directly to the residual stream at a ⊕ node.

The weight matrix Wdown has shape [dmodel, dhidden] — opposite of the gate and up matrices which are [dhidden, dmodel]. That shape means it must compress information. The learning problem is to decide what to keep: coordinates in the hidden space that carried useful, gate-selected signal should project onto the residual-stream dimensions most able to absorb them. Dimensions that were quieted by the gate contribute little regardless of Wdown, because SwiGLU has already driven them toward zero.

In the diagram the down projection is the single arrow that leaves the SwiGLU node and arrives at the ⊕ adder on the residual backbone. The output it delivers has exactly the same shape as the backbone — that is what makes the residual addition valid. The feed-forward block therefore forms a closed loop: branch up from the stream, widen, gate, compress, add back.

Down projection output: a model-dimension vector returned to the residual stream. Amber cells add positively; teal cells subtract — the net effect accumulates in the backbone.