RoPE — rotary positional encoding

Injecting position into attention by rotating query and key vectors, so relative distance appears as a phase difference.

Without positional information, a transformer is permutation-invariant: it cannot tell whether "cat sat mat" was written left-to-right or in any other order. RoPE (Rotary Position Embedding) solves this by rotating the query and key vectors just before the dot-product attention score is computed. It acts on each head independently: a head's slice — for example dhead = 16 dimensions (a dim of 64 split across 4 heads) — is treated as 8 pairs. Pair j is rotated by angle θⱼ = position × base^(−2j/dhead), where base is a fixed constant (often 10 000). The first pairs use large angles (fast rotation, fine-grained position); the later pairs use tiny angles (slow rotation, coarse, long-range structure).

The key geometric property is that the dot product q · k after rotation depends only on the relative position (posq − posk), not on absolute positions. Attention automatically learns "how far apart are these two tokens" rather than "what absolute slot are they in". This generalises cleanly to sequences longer than those seen in training — you can extrapolate positions the model was never trained on and the angle formula still produces valid rotations. No extra learned parameters are needed: the encoding is entirely determined by the formula.

In the diagram, the RoPE node sits on the Q and K branches just before the attention matrix is computed. The rope-dial below shows the 8 dimension-pairs as clock faces: each dial is rotated by the current token's position angle for that frequency (the same 8-pair rotation is applied to every head). Fast-spinning pairs (left side) are sensitive to nearby context; slow-spinning pairs (right side) encode whether two tokens are many steps apart. Watch the dials change as the position index increases — pairs on the left complete many full rotations while pairs on the right have barely moved.

The 8 dimension-pairs of a head, rotated by this token's position. Left dials spin fast (fine position); right dials spin slow (coarse). The dot product between Q and K inherits only the relative phase, not the absolute angle.

Related: Attention · The transformer block · Embedding