Weighting the values

Mixing the value vectors in proportion to the attention weights — the payoff of attention.

By this point attention has a row of weights: for the current token's query, a number per earlier position saying how much to attend there, all non-negative and summing to one (the softmax output). The final step spends those weights. Each attended position contributed a value vector V; the output is their weighted sum, context = Σt weightt · V_t. A position with weight 0.6 contributes 60% of its value; a position with weight 0.01 barely registers. The result is one vector — a blend of the past, mixed by relevance.

This is why the weights and the values play different roles. The query–key dot products only decide the proportions; the values are the actual content being averaged. Change what a token attends to and you re-mix the same values into a different blend; change the values and the same weights carry different information forward. Because the weights sum to one, the context is a true weighted average — it can't blow up, it just shifts toward whichever positions won the attention.

Each head does this independently over its own slice, and the per-head contexts are concatenated and passed through the output projection Wo before being added back to the residual stream. The grid below shows the weights one head places over positions; read a bright cell as "this much of that position's value flows into the result."

Related: Softmax weights · Query, Key, Value · Multi-head