The causal mask

Blocking future positions so each token's prediction depends only on what came before it.

The causal mask is a triangular pattern of −∞ values added to the score matrix before softmax. Position i can only attend to positions j ≤ i — itself and everything to its left. Scores for positions j > i are set to −∞, so exp(−∞) = 0 and they vanish from the softmax sum entirely. Every row ends up as a distribution over a strictly non-future window. Without this mask, the model could look ahead at the answer it is supposed to predict — training would be trivially solvable and generation would break because future tokens do not yet exist.

In the standard formulation the mask is an additive matrix: scores_masked = scores + mask, where mask[i][j] = 0 when j ≤ i and −∞ otherwise, so exp(−∞) = 0 zeros the future. A streaming implementation reaches the same result more directly: when scoring the token at position, it sizes the score buffer to position + 1 and only ever loops over keys 0 … position — the future is never scored in the first place, so there is nothing to mask away. Either way the effect is identical, and during autoregressive generation — when the sequence is extended one token at a time — the current token is always the newest row, so no special handling is needed.

In the diagram, the narrow grey cap on the right edge of each head's score grid is the masked region — future positions for the token currently being generated. After softmax those cells contribute nothing. The weights grid (right) shows the effect: all attention is concentrated on the visible past, and the masked column is empty.

The grey cap at the right edge of each row is the causal mask — those positions receive −∞ before softmax, so they get zero weight. Every head sees only the past and present.