Parameters

The learned numbers inside the model — one count that describes its size.

A parameter is a single number the model learned during training. The model's weights are matrices of parameters; training adjusts them to minimize error on the text it saw, and generation reads them to predict tokens. The total number of parameters is how people measure model size: a "7 billion parameter model" has 7 billion learned numbers. A teaching-sized model has only a tiny fraction of that — on the order of a hundred thousand — small enough to stay fast and fully readable; the playground shows the exact count for whatever model it is running.

Parameters live in several places. The token embedding layer has one weight matrix of shape [vocabSize, dim] — one row per token in the vocabulary, so embedding token 42 reads that token's row (a vector of dim parameters). In each attention sub-layer, there are weight matrices that project the input into queries, keys, and values, and another that projects the attention output back. In each feed-forward sub-layer (a SwiGLU network), there are three weight matrices: two (Wgate and Wup) project the input up into a larger hidden space, and one (W_down) projects the result back down. The final norm and output projection each contain parameters too. Every weight matrix is a table of learned numbers; every layer stacks more of them; every new layer adds tens of thousands of parameters to the total. Parameters never change during generation — they were fixed at training time — but they are the only things in the network that truly matter. Everything else (the intermediate vectors, the attention scores, the logits) are computed from the parameters and the input tokens.

In the diagram, weight matrices appear as dashed boxes below the residual stream. When you hover over an operation, the count of parameters in that operation appears in the header. Expand any layer and you will see all its parameter matrices labeled by their shape and purpose. The visual hierarchy reminds you that parameters are stored and fixed, while the activations (the flowing vectors) are computed on the fly for each token.