The generation loop

The autoregressive loop: predict the next token, append it, and repeat until stopping.

Generation is a feedback loop, not a one-shot prediction. The model takes the prompt and predicts a distribution over the next token (via the pipeline: embedding → attention → feed-forward → logits → softmax). The sampler picks one token from that distribution (greedy, random, top-k, top-p, or by temperature). That token is appended to the sequence. The model then reads the full growing sequence (prompt + all generated tokens so far) and predicts the next token, picks it, appends it, and repeats. This is autoregressive: each prediction depends on the history of previous predictions.

The loop runs until a stopping condition is met. The simplest is max length — you set a maximum number of tokens (e.g., "generate up to 100 tokens"), and the model stops when it reaches it. Another is an end-of-sequence token: the tokenizer reserves one id to mark "I am done", and if the sampler picks that token, generation halts. You can mix both — generate until max length or until the model says it is finished, whichever comes first. Without either, the loop would run forever.

In the diagram, you press the Generate button and the sequence grows. The token chips accumulate on the right as new tokens are appended. Each one flows through the full stack, a new logit vector is computed, the sampler picks based on the current temperature / top-k / top-p settings, and the token appears on the diagram. If you watch closely you see each prediction reads all previous tokens — that is causal attention: the model looks at context but never cheats by looking ahead. The loop is deterministic with greedy sampling (same sequence every time) but random otherwise; that randomness is where variety and creativity come from.

Related: The prompt · The token · Sampling