The feed-forward network

The per-token MLP applied after attention — here the SwiGLU variant.

In this section

• Gate and up — The two parallel projections that expand into the wide hidden layer.
• SwiGLU — SiLU(gate) ⊗ up — the gated non-linearity at the heart of the feed-forward.
• Down projection — Projecting the wide hidden layer back down to [dim].
• Hidden dimension — What the feed-forward width is, and why it's larger than the model dimension.