Autoregressive generation

Sampling one token at a time and feeding it back as input to generate the next, growing the sequence step by step.

Autoregressive means the model generates by feeding its own output back as input — "auto" (self) + "regressive" (self-predicting). Here is how it works: you start with a prompt (some initial tokens), run the forward pass on the last token in the prompt, sample one token from the resulting logits, append that token to the sequence, and repeat. Each iteration adds one token; each new token sees all previous tokens (the prompt plus everything generated so far). This is why early tokens in your generation influence later ones — each forward pass conditions on the full history, and the sampler picks the next token randomly (or greedily, or via top-k, etc.). The sequence grows by one token per forward pass, so generating N tokens takes N forward passes.

This approach is simple but powerful. The model has no "plan" for what to generate; it only looks at the history and picks the next most likely token. Yet because it was trained on real text, those one-token-at-a-time decisions often produce coherent paragraphs. It is also the only way to do inference: the model was designed to predict one token given a context, so generating is just repeating that prediction, feeding in the new context each time. There is no way to generate all N tokens in parallel (unlike some other architectures); you must generate left-to-right, one token at a time, because each token depends on all the ones before it.

In the diagram, you see this loop played out. The prompt occupies the first few positions of the residual stream. After the pulse reaches the output and samples a token, that token is appended to the sequence (you see it light up in the token rail on the left), and the pulse re-enters the embedding layer for the next forward pass. This time, the attention heads read not just the prompt but the prompt plus the one token you just generated. The process repeats until you reach the maximum token count or a stopping condition. Press Pause after a few generations and count the tokens — you will see the sequence grow by exactly one token per forward pass.

Related: The forward pass · Sampling · The generation loop