Multi-head attention

Running H independent attention operations in parallel so the model can attend to different things at once.

Multi-head attention splits the residual-stream vector into H equal subspaces and runs the full scaled-dot-product attention separately in each one. Each head has its own learned projection matrices Wqh, Wkh, Wvh that carve out its slice of query, key, and value space. The heads operate completely independently — different heads learn to attend to different relationships. One head might focus on syntactic proximity, another on semantic similarity, another on the most recent token. After each head produces its output vector, all H outputs are concatenated and projected back to the full model dimension via Wo.

The dimension of each head is dhead = dmodel / H. The total parameter count is the same as a single-head model — each head is cheaper, but there are more of them. The crucial property is that the heads do not share their internal projections, so they genuinely attend to different patterns. Any single attention head can only produce a single weighting over positions; with H heads the model can maintain H simultaneous and distinct weightings and blend them linearly via W_o.

In the playground, the residual stream runs as a horizontal neutral backbone. Attention branches upward to compute the per-head Q, K, V projections (with RoPE applied to Q and K for positional encoding), runs the scaled dot-product operation with causal mask, then the per-head outputs are concatenated and added back to the stream at the ⊕ residual add. The figure below shows all four heads side-by-side — notice how their score patterns differ: one peaks at the last position (recency), one anchors at position 0 (first-token bias), one spreads evenly (diffuse), one is bimodal.

Four independent heads (H0–H3) attending over six positions. Each row is one head's pattern — recency, first-token bias, diffuse, and bimodal — demonstrating how heads specialise into distinct attention behaviours.