Scaled dot-product attention

Scoring how well each query matches every key — the core operation that lets tokens reach across the sequence.

Scaled dot-product attention is the arithmetic heart of the attention mechanism. For each token, the model produces a query vector Q and compares it against a key vector K for every past position. The comparison is a dot product: Q · Kᵀ. A high dot product means the query and key are aligned — this position is relevant. The result is one raw score per past position, forming a row in the score matrix.

The scaling step divides every score by √dk, where dk is the dimension of the key vectors. Without it, dot products grow large as dk increases: if the components of q and k have unit variance (the paper's assumption), their dot product q·k = Σᵢ qᵢkᵢ has variance dk, so its typical magnitude scales with √dk. Large scores push softmax into regions where its gradient nearly vanishes — the model stops learning. Dividing by √dk rescales the score back to roughly unit variance, where softmax stays sensitive. The full operation is: scores = Q · Kᵀ / √d_k.

In the diagram below, the left grid is the raw score matrix — amber cells are positive scores (query and key point in the same direction), teal cells are negative. Each row is one head's view; each column is one past position. The right grid is the same data after softmax, so each row sums to 1. The narrow grey cap on the right edge of each grid is the causal mask: future positions are forced to −∞ before softmax, so they receive zero attention weight.

Left: raw Q·Kᵀ/√d_k scores (amber = positive, teal = negative). Right: after softmax, each row sums to 1. The grey cap is the causal mask — future positions are zeroed out.