Mixture-of-Experts

Holding many feed-forward networks but running only a few per token.

A dense model runs every weight for every token. A Mixture-of-Experts (MoE) layer changes one thing — it replaces the single feed-forward network with a bank of many experts (each an ordinary feed-forward network) plus a small router that, for each token, picks only the top-k experts to actually run. Attention usually stays dense; it's the feed-forward that goes sparse.

That single change decouples two numbers a dense model keeps equal: capacity (all the experts you store) from compute (the few that fire per token). It's how a model can be "30B total, 3B active" — large in memory, small in per-token cost. The pages below build the idea up one piece at a time, each standalone:

The router — the gating network that scores experts and picks the top-k.
An expert — why an "expert" is just the feed-forward network you already know.
Sparse activation — running only k / E of the experts, and the two parameter counts it creates.
Load balancing — keeping the router from overusing a few experts, and the auxiliary loss that helps.

For how MoE compares to the dense base model, and how its parameter counts are named, see Dense vs Mixture-of-Experts and Active vs effective parameters.

The real routing here is illustrated with seeded weights (the decoupled demo in src/variants/), so the figures show the mechanism — route, fire a sparse few, renormalize, balance — not trained quality.

Sparse activation

Running only a few of the experts per token — the source of MoE's efficiency.

Sparse activation is the payoff of routing. A dense layer is dense because every weight participates in every token: 100% of the FFN runs every time. An MoE layer is sparse because the router fires only the top-k of its E experts, so each token touches just k / E of the expert weights. With 8 experts and top-2 routing, that's a quarter of the expert compute per token — the other six experts sit idle for that token and cost nothing.

This is why MoE has two parameter counts that a dense model collapses into one. Total parameters grow with every expert you add — that's the model's capacity, what sits in memory. Active parameters, the weights actually multiplied to produce a token, stay pinned to the k experts that fire. Add experts and total climbs while active barely moves; the ratio k / E is exactly how sparse the layer is. Crucially, sparsity buys compute, not memory: every expert must still be loaded to be routable, so a sparse model is cheaper to run per token but not smaller to store.

The figure makes the dial concrete: at top-k = 2 only two of eight experts light up — 25% active — and the footer tracks the percentage as you change k. Slide k all the way to 8 and every expert fires: the layer is fully dense again, with no compute saved. Sparse activation lives entirely in that gap between a small k and a large E.

Only the top-k experts fire; the rest are idle at zero cost. k/E is the fraction of expert compute spent — small k over large E is what 'sparse' means.

The headline you see on modern models — "30B total, 3B active" — is this idea stated as numbers: a large pool of experts for capacity, a tiny fraction switched on per token for speed.