Mixture-of-Experts

Holding many feed-forward networks but running only a few per token.

A dense model runs every weight for every token. A Mixture-of-Experts (MoE) layer changes one thing — it replaces the single feed-forward network with a bank of many experts (each an ordinary feed-forward network) plus a small router that, for each token, picks only the top-k experts to actually run. Attention usually stays dense; it's the feed-forward that goes sparse.

That single change decouples two numbers a dense model keeps equal: capacity (all the experts you store) from compute (the few that fire per token). It's how a model can be "30B total, 3B active" — large in memory, small in per-token cost. The pages below build the idea up one piece at a time, each standalone:

The router — the gating network that scores experts and picks the top-k.
An expert — why an "expert" is just the feed-forward network you already know.
Sparse activation — running only k / E of the experts, and the two parameter counts it creates.
Load balancing — keeping the router from overusing a few experts, and the auxiliary loss that helps.

For how MoE compares to the dense base model, and how its parameter counts are named, see Dense vs Mixture-of-Experts and Active vs effective parameters.

The real routing here is illustrated with seeded weights (the decoupled demo in src/variants/), so the figures show the mechanism — route, fire a sparse few, renormalize, balance — not trained quality.

An expert

One of the many parallel feed-forward networks an MoE layer chooses between.

An expert sounds exotic, but it is the most familiar piece of the model: a single feed-forward network, identical in shape to the one a dense block runs. In the SwiGLU form this project uses, that's three matrices — a gate projection, an up projection, and a down projection — wrapped around the SwiGLU activation. An MoE layer is just many of these FFNs side by side (here 8 of them) instead of one, plus a router that decides which ones run.

Because every expert has the same architecture as the dense FFN, the maths inside a fired expert is exactly the feed-forward maths covered elsewhere: down · (silu(gate · x) ⊙ (up · x)). Nothing about being "an expert" changes how the network computes — what changes is that the layer holds several of them and runs only the k the router selected. The experts do not specialize by hand; in a trained model the router and the experts learn together, and experts drift toward handling different kinds of tokens. (The seeded experts behind the figure aren't trained, so they carry no such meaning — they illustrate the structure, many parallel FFNs, not learned specialties.)

The figure shows the rack: each bar is one expert, and the lit ones are the experts actually running for the current token. Switch the input or change how many may fire to see the same pool of experts used differently from token to token — that reuse of a large pool by a small active subset is what an expert is for.

Each bar is one expert — a full SwiGLU FFN. An MoE layer keeps many in parallel; the router lights only the few that run for this token.

So "expert" is a routing word, not an architecture word: the expert is the ordinary feed-forward network you already know, multiplied and made selectable.

Related: SwiGLU · The feed-forward network · The router · Sparse activation