Dense vs Mixture-of-Experts

Whether every weight runs for each token, or a router activates only a chosen few.

In a dense transformer — Llama, GPT-2/3, Mistral 7B — every weight participates in every token's forward pass. The feed-forward network is a single set of matrices (here Wgate, Wup, W_down) that every token flows through. So the compute per token scales with the whole model: its active parameters equal its total parameters. Nothing is skipped, which is exactly why a dense model is the most straightforward to follow.

A Mixture-of-Experts (MoE) model replaces that single feed-forward with a bank of N independent feed-forwards — the experts — plus a small router (a gating linear layer). For each token the router scores the experts, picks the top-k (often 2 of 8, or 8 of 128), runs only those, and blends their outputs weighted by the router's scores. The rest are skipped for that token. Attention usually stays dense; it is the feed-forward that goes sparse.

The point is to decouple capacity from compute. The total parameters (all experts) hold the model's knowledge; the active parameters (the top-k that actually run) set the cost per token. So you can add experts — more capacity — almost free in compute. Mixtral 8×7B (8 experts, top-2) is ~47B total but only ~13B active; Qwen3-30B-A3B is 30.5B total / 3.3B active (128 experts, 8 active); Gemma-4's 26B-A4B is 25.2B total / 3.8B active. Each runs at roughly the speed of a small model while holding the capacity of a large one.

Add experts and total parameters (amber, memory) climb while active parameters (teal, per-token compute) barely move — they track only the top-k that fire. At 1 expert it's a dense layer: total = active.

The catch: MoE saves compute, not memory. You must still store every expert, so a 26B MoE occupies ~26B worth of memory even though only ~4B compute per token. It also needs an auxiliary load-balancing loss (so the router doesn't collapse onto a few favourite experts) and more complex distributed serving (experts often live on different devices). A dense model keeps every weight real and active — the easiest case to trace — which is why it's the natural starting point before adding the routing machinery above.