Logits

The output projection turns the final 64-dimensional vector into one raw score per vocabulary token.

After the final RMSNorm, the residual stream is a single 64-dimensional vector. Logits are produced by multiplying that vector by the output-projection weight matrix — shape [27, 64] — yielding 27 raw scores, one per token: logits = W_out · x. Each score is an unnormalised log-probability: a higher number means the model assigns more relative likelihood to that token being next. There is no activation function here; the scores come out of a plain matrix-vector product and can be any real number, positive or negative.

The output projection Wout is its own learned matrix — a separate [vocabSize, dim] array from the embedding table (these weights are often not tied, though some models do tie them). Geometrically the projection still measures alignment: each row of Wout is a direction in the 64-dimensional space, and a logit is the dot product of the final hidden vector with that row. A high logit for token i means the final vector points strongly along token i's output direction — the model has steered the residual stream toward the token it wants to emit.

In the diagram, the output projection is the last named operation on the residual stream, immediately following the final-norm node. The logit-bars figure below shows all 27 scores as horizontal bars; amber marks positive scores, teal marks negative. Before any softmax, the bars can span a wide range — the gap between the top score and the rest determines how confident the prediction is. Sampling strategies (temperature, top-k, top-p) then reshape this distribution before picking.

27 raw output scores — one per vocabulary token. Amber = positive, teal = negative. The tallest amber bar is the greedy pick; sampling strategies then draw from the softmax of these values.

Related: Final norm · The output token · Temperature