Mixture-of-Experts

Holding many feed-forward networks but running only a few per token.

A dense model runs every weight for every token. A Mixture-of-Experts (MoE) layer changes one thing — it replaces the single feed-forward network with a bank of many experts (each an ordinary feed-forward network) plus a small router that, for each token, picks only the top-k experts to actually run. Attention usually stays dense; it's the feed-forward that goes sparse.

That single change decouples two numbers a dense model keeps equal: capacity (all the experts you store) from compute (the few that fire per token). It's how a model can be "30B total, 3B active" — large in memory, small in per-token cost. The pages below build the idea up one piece at a time, each standalone:

The router — the gating network that scores experts and picks the top-k.
An expert — why an "expert" is just the feed-forward network you already know.
Sparse activation — running only k / E of the experts, and the two parameter counts it creates.
Load balancing — keeping the router from overusing a few experts, and the auxiliary loss that helps.

For how MoE compares to the dense base model, and how its parameter counts are named, see Dense vs Mixture-of-Experts and Active vs effective parameters.

The real routing here is illustrated with seeded weights (the decoupled demo in src/variants/), so the figures show the mechanism — route, fire a sparse few, renormalize, balance — not trained quality.