SwiGLU

A gated activation that multiplies two parallel signals — one shaped by SiLU, one raw — to let the network suppress irrelevant dimensions.

SwiGLU is the non-linearity at the centre of the feed-forward sub-layer. Given the two hidden-dimension vectors produced by the gate and up projections, it computes SiLU(gate) ⊗ up: element-wise multiply the SiLU-activated gate by the un-activated up vector. SiLU(x) = x · σ(x) — sigmoid-weighted identity — is smooth, non-monotonic, and nearly linear for positive values while gently suppressing negatives.

The gating structure is the point. A plain ReLU(up) would set every negative dimension to zero indiscriminately; the separate gate path learns which dimensions should survive for this particular input. When SiLU(gate[i]) is close to zero the corresponding up[i] contribution is silenced; when it is close to its input value the up-stream activations pass through unmodified. The network therefore learns to route information selectively rather than applying a fixed hard threshold.

In the diagram the SwiGLU node sits between the two upward projections (gate and up) and the single down projection. Its output vector has the same hidden-dimension width as its two inputs; the element-wise multiplication happens in-place before the down projection reduces it back to model dimension. The heat strip below shows the result: amber cells (positive) carry the surviving signal; teal cells (suppressed negatives) indicate dimensions the gate has quieted.

SwiGLU output: amber dimensions carry forward, teal dimensions were gated down. Each cell is one hidden-dimension slot.