Top-p (nucleus)

Restricting to the smallest set of tokens whose probability covers p.

Top-p, also called nucleus sampling, trims the candidate set by probability mass rather than by count. Sort the tokens from most to least likely, walk down the list adding up their probabilities, and stop as soon as the running total reaches p (say 0.9). That smallest group — the "nucleus" — is the candidate set; renormalise it to sum to 1 and sample from it.

The advantage over a fixed cutoff is that the shortlist resizes itself to the model's confidence. When the model is sure, a single token may already cover 90% of the mass, so the nucleus is tiny and the choice is nearly deterministic. When the model is genuinely uncertain, many tokens are needed to reach p, so the nucleus grows and more options stay in play. The threshold adapts where a fixed k cannot.

In practice top-p is a popular default because it gives controlled variety without hand-tuning a count per prompt. It also composes with the other dials: temperature can reshape the distribution first, and top-p then decides how much of that reshaped mass to keep.

Related: Top-k · Temperature · From logits to probabilities