Mixture-of-Experts

Holding many feed-forward networks but running only a few per token.

A dense model runs every weight for every token. A Mixture-of-Experts (MoE) layer changes one thing — it replaces the single feed-forward network with a bank of many experts (each an ordinary feed-forward network) plus a small router that, for each token, picks only the top-k experts to actually run. Attention usually stays dense; it's the feed-forward that goes sparse.

That single change decouples two numbers a dense model keeps equal: capacity (all the experts you store) from compute (the few that fire per token). It's how a model can be "30B total, 3B active" — large in memory, small in per-token cost. The pages below build the idea up one piece at a time, each standalone:

The router — the gating network that scores experts and picks the top-k.
An expert — why an "expert" is just the feed-forward network you already know.
Sparse activation — running only k / E of the experts, and the two parameter counts it creates.
Load balancing — keeping the router from overusing a few experts, and the auxiliary loss that helps.

For how MoE compares to the dense base model, and how its parameter counts are named, see Dense vs Mixture-of-Experts and Active vs effective parameters.

The real routing here is illustrated with seeded weights (the decoupled demo in src/variants/), so the figures show the mechanism — route, fire a sparse few, renormalize, balance — not trained quality.

The router

The small network that decides which experts handle each token.

A router (or gate) is what turns a pile of expert networks into a Mixture-of-Experts. Where a dense layer runs its one feed-forward network on every token, an MoE layer holds many expert FFNs and, for each token, the router picks only a few of them to actually run. The router itself is tiny: a single matrix that projects the token's vector to one score per expert.

The decision is three plain steps. First, score every expert and turn the scores into a distribution: probs = softmax(router · x) — one probability per expert. Second, keep only the top-k experts (commonly k = 2) and discard the rest; those discarded experts do no computation for this token. Third, renormalize the surviving probabilities so they sum to 1 — these become the gate weights that blend the chosen experts' outputs: gatei = pi / Σ_chosen p. That is the entire mechanism; everything else in the layer is ordinary feed-forward maths.

In the figure, each bar is one expert's router probability for the current input. Switch the input and the tall bars move — a different token routes to different experts. Drag top-k and watch experts switch on or off: at k = 2 only two fire (25% of the expert compute) and their gates renormalize to 100% between them; push k up and more experts light up until, at k = 8, it's dense again. The experts here are generic (e0…e7) on purpose — in a trained MoE the router learns to send related tokens to the same experts, so experts specialize; the seeded weights behind this demo carry no such meaning, so what's real to read here is the routing operation, not any expert's "topic".

softmax(router·x) scores every expert; only the top-k fire and their gates renormalize to sum 1. Lower k = sparser = less compute per token; k = E is just a dense layer again.

The router is the one new idea MoE adds, and it's what makes the model's two parameter counts diverge: total weights grow with every expert added, but the active weights per token stay fixed at the k that fire. That gap — capacity without proportional compute — is the whole point of the design.