Weight layout

How this project stores weight matrices so matrix-vector math reads naturally: shape is [outDim, inDim].

Every operation in the transformer computes y = W · x: a weight matrix W multiplied by an input vector x to produce an output y. Mathematically this is unambiguous, but when storing a matrix in a flat array, you must choose an order: do you store row-by-row or column-by-column? This project uses row-major, output-first layout: the matrix has shape [outDim, inDim], meaning it has outDim rows and inDim columns. Row i, column j contains the weight that multiplies input element j and feeds into output element i. This means a single dot product — one row of W times the vector x — produces one element of the output y.

This layout is chosen for readability. When you look at the code and see matVec(w.wq, x) (computing queries from a layer's weights w), the weight matrix shape immediately tells you: wq has shape [dim, dim] = [64, 64], so multiplying it by an input of shape [dim] gives output of shape [dim] — the full query vector, which attention then slices into per-head headDim-length pieces. No mental translation needed. The code and the math match. If the layout were transposed, the same line would compute y = x · W, which is less intuitive and requires you to think about why the dimensions are backwards. By always storing [outDim, inDim], every matrix-vector product reads as y = W · x directly from the code.

In practice, this means that when you load a weight matrix from disk and want to read the weights that feed into output element 7, you read row 7 from the matrix. The first element of row 7 is the weight that scales input element 0, the second element is the weight for input element 1, and so on. A matrix of shape [27, 64] — the output projection here, [vocabSize, dim] — has 27 rows (one per output element, one per vocabulary token) and 64 columns (one per input element); multiplying it by a vector of 64 elements yields a vector of 27. When you see a weight matrix in the diagram, its shape label is always [outDim, inDim] — you can trust that convention to decode its purpose immediately.

Related: Weights · Parameters · Forward pass