Overview

A neural network that reads tokens one at a time and predicts the next one, learned from text.

A transformer is a machine-learning model that operates on sequences. The kind these lessons focus on is decoder-only, meaning it reads text left-to-right and predicts what comes next — the same process you use when you type and your phone suggests the next word. The "decoder" name comes from the machine-learning genealogy: the model is trained to decode (generate) the next token given a context, rather than encode an image or translate between languages. It does this by reading weights — learned numbers stored in memory — and applying the same sequence of operations to every token.

The transformer's core idea is attention: when the model sees token 42 in position 7, it doesn't just guess independently; it looks back at all previous tokens (positions 1 through 6), computes a relevance score for each one, and blends their information to decide what comes next. This happens in parallel across many "heads" (separate attention channels) and is followed by a feed-forward network that refines the blend. The model repeats this pattern N times, in N stacked layers, allowing later layers to build on earlier ones. After all layers, a final projection produces one score (a logit) for each token in the vocabulary; the token with the highest score is the model's best guess for what comes next.

In the diagram, the residual stream is the horizontal spine — the single vector that flows left-to-right through the network. Attention and feed-forward operations branch upward from it, apply their math, and add their output back to the stream (a residual connection) so information can flow cleanly all the way from the input token to the output logits. The pulse marks the active stage as you generate: it steps through embedding, then the first layer, then the second, and so on, until it exits the final projection and becomes your next token. Press Pause and hover over any operation to see the real numbers flowing through.