The forward pass

One complete computation from input token to output logits: embedding → N layers → final norm → projection.

A forward pass (or forward computation) is what happens when the model processes a single token. It starts with a token id (an integer from 0 to vocabSize-1), feeds it through the embedding layer to get a dense vector, pipes that vector through all N transformer blocks in sequence, applies a final normalization, and ends with an output projection that produces one score (logit) for each possible next token. The output is a vector of length vocabSize; the largest element tells you which token the model thinks should come next.

Mathematically, a forward pass is a composition of functions: y = project(norm(blockN(...block1(block_0(embed(tokenId)))...))). Each block is itself a composition of two sub-operations (attention and feed-forward), each with its own weight matrices. The entire computation is deterministic — the same input token at the same position in the sequence always produces the same output logits (assuming no randomness in any layer). The forward pass is the spine of the transformer; training computes it millions of times to adjust weights, and generation computes it once per token to predict what comes next.

In the diagram, the forward pass is the pulse traveling left-to-right. It enters from the left as a token id, undergoes embedding, steps through each layer (the residual stream evolves at each step), passes through final normalization, exits the output projection, and produces a bar chart of logits on the right. Press Step in the playground to advance the pulse one stage at a time and watch the hidden state (the residual stream) transform as it moves through each operation. The forward pass is the journey the diagram illustrates.

Related: Overview · Transformer block · Layer stack