Down projection

A single linear map that compresses the wide hidden-dimension vector back to model dimension and deposits the result into the residual stream.

The down projection is the final step of the feed-forward sub-layer. After the gate and up projections have expanded the residual stream to the hidden dimension and SwiGLU has selectively combined them, a single weight matrix W_down maps the resulting d_hidden vector back to d_model: output = W_down · swiglu_out. This is a pure linear projection — no activation, no gating — and the result is added directly to the residual stream at a ⊕ node.

The weight matrix W_down has shape [d_model, d_hidden] — opposite of the gate and up matrices which are [d_hidden, d_model]. That shape means it must compress information. The learning problem is to decide what to keep: coordinates in the hidden space that carried useful, gate-selected signal should project onto the residual-stream dimensions most able to absorb them. Dimensions that were quieted by the gate contribute little regardless of W_down, because SwiGLU has already driven them toward zero.

In the diagram the down projection is the single arrow that leaves the SwiGLU node and arrives at the ⊕ adder on the residual backbone. The output it delivers has exactly the same shape as the backbone — that is what makes the residual addition valid. The feed-forward block therefore forms a closed loop: branch up from the stream, widen, gate, compress, add back.

Down projection output: a model-dimension vector returned to the residual stream. Amber cells add positively; teal cells subtract — the net effect accumulates in the backbone.