The residual vector

A single floating-point vector that every layer reads from and writes back into — the backbone of the whole network.

The residual stream is the d_model-dimensional vector that travels with each token position from the embedding lookup all the way to the final logit projection. It is not a metaphor: it is one concrete array of numbers (say 64 floats) that lives in memory for the entire forward pass. Every sub-layer — attention, feed-forward, normalization — takes this vector as input and returns a delta that gets added back. Nothing is discarded; the vector simply accumulates edits.

The values inside it do not correspond to hand-labelled concepts. After training, different dimensions specialise to track different features — syntactic role, semantic category, positional context — but the encoding is distributed and emergent, not designed. What you can see is whether a dimension is currently positive or negative (the amber/teal diverging colours in the diagram below) and how large its activation is (colour height). Early in the network most dimensions are quiet; by the final block the pattern is dense with information specific to the current context.

One useful way to read the vector at any stage is to project it straight into vocabulary space using the output weight matrix — this is the logit-lens idea — and ask "what token would the model predict if we stopped here?" Early stages give vague, often wrong answers; late stages converge on the right one. The sharpening of that guess, layer by layer, is visible evidence that the residual stream is accumulating the right information.

Related: Residual add · Logit lens