Softmax over attention scores

Turning a row of raw scores into a probability distribution so each token can blend the past.

Softmax converts a row of unbounded real-valued scores into a set of non-negative weights that sum exactly to 1. For attention, each row in the score matrix becomes an independent distribution: weightsi = exp(scorei) / Σj exp(scorej). The highest score dominates — the gap between scores is preserved and amplified — but no weight is ever zero (unless a position is explicitly masked), and every position gets at least some share of attention.

The numerically stable version subtracts the row maximum before exponentiating: exp(score_i − max). This prevents floating-point overflow without changing the result, because the constant cancels in the numerator and denominator. After the causal mask sets future positions to −∞, those terms become exp(−∞) = 0, cleanly excluding them from the sum. The weights are then used to blend the Value vectors: output = weights · V.

In the figure, the left grid holds the raw Q·Kᵀ/√d_k scores and the right grid is the result after softmax. Watch the contrast: a moderately high raw score for one position — say the "recency" head's peak on the last column — becomes an overwhelming weight on the right, because exponentiating magnifies differences. An evenly spread head on the left produces a nearly uniform right grid.

Left: raw scaled scores. Right: after softmax, every row sums to 1. A dominant score on the left becomes a near-1 weight on the right; a flat row stays flat.