Attention

The mechanism a token uses to read information from earlier tokens.

In this section

• Query, Key, Value — Three learned projections of the same vector: what a token seeks, offers, and passes on.
• Multi-head attention — Splitting the vector into H subspaces that each attend independently.
• Scaled dot-product — Q·Kᵀ/√d_k: scoring how well a token's query matches each earlier key.
• Softmax — Turning a row of raw scores into a probability distribution over positions.
• Weighting the values — Using the attention weights to take a weighted average of the Values (·V).
• The causal mask — Why a token can never attend to the future — scores are set to −∞ before softmax.