Query, Key, Value

Three learned projections of the same vector — what a token seeks, offers, and carries.

Attention begins by turning one input vector into three. The token's hidden state x (an RMSNorm'd vector on the residual stream) is multiplied by three separate learned weight matrices to produce a query Q = Wq·x, a key K = Wk·x, and a value V = Wv·x. Same input, three different lenses on it. None of these is more fundamental than x — they are just x viewed through three learned projections, and the matrices Wq, Wk, Wv are what training shaped.

The three roles are distinct. A token's query is what it is looking for in earlier tokens; a token's key is what it advertises about itself to others' queries; its value is the information it will contribute if it gets attended to. Attention compares one token's query against every earlier token's key to decide how much to look, then mixes the corresponding values. So Q and K only ever meet as a dot product (a match score), while V is the payload that actually flows onward.

Often the projections preserve the full width — x, Q, K, and V are each dim-length vectors (e.g. 64) — and are then sliced across heads downstream. The figure below shows the fan-out: one input on the left, the three real projection outputs on the right, each produced by its own weight matrix. Each is just a tensor of numbers; what makes them queries, keys, and values is only how attention uses them next.

One input x → three learned projections Q = Wq·x, K = Wk·x, V = Wv·x. Same vector, three lenses: what the token seeks (Q), what it advertises (K), and what it passes on (V).

Related: Scaled dot-product · Weighting the values · Multi-head