Weights

The learned matrices that define the model's behavior — the only numbers that training changes.

Weights are the learned parameters organized into matrices. They are the essence of a trained model: if you copied all the weights from one model to another identical architecture, the second one would behave identically. Every weight is a floating-point number that training optimized (via backpropagation and gradient descent) to make the model's predictions match real text. After training stops, weights are frozen — they never change again during generation.

There are two types of numbers in the model: weights (learned parameters, stored in files) and activations (computed values, created fresh for each token). A weight matrix might be [64, 64] (an attention projection) or [128, 64] (a feed-forward projection), stored in the model file, never changing. An activation might be the residual vector of shape [64] flowing through a layer, computed on the fly from the input token and the weights, different for every input. When you run the model, the code reads weights from memory and combines them with the current activation using matrix math (matrix-vector products, element-wise operations, normalizations) to compute the next activation. This repeats until you get logits, and sampling picks a token from those logits. The fixed weights are what make the model do what it does; the flowing activations are just the consequence.

In the diagram, weights are depicted as dashed boxes (distinguishing them from the solid operations) and appear below the residual stream. Activations flow along the stream itself — the solid vectors that change with every new token. Hover over a dashed box to see the weight matrix's shape and a sample of its learned values. The visual separation reminds you which numbers are part of the model's identity (weights, stored forever) and which are transient (activations, computed, discarded).

Related: Parameters · Weight layout · Forward pass