Top-k

Restricting the choice to the k most-likely tokens.

Top-k sampling trims the field before the model picks. It sorts the tokens by probability, keeps only the top k of them, throws the rest away, and renormalises those k back into a distribution that sums to 1 — then samples from that shortlist.

The point is to get the variety of random sampling without its worst failure mode. Pure sampling from the full vocabulary can occasionally land on a wildly unlikely token and derail the text; capping the candidates at k (say 40) keeps the choice random among plausible options only. A small k is conservative and close to greedy; a large k approaches sampling from the whole distribution.

Its limitation is that k is a fixed count, blind to shape. When the model is very confident, a fixed k still drags in k−1 no-hope tokens; when it's genuinely torn between many options, the same k may cut off good ones. That mismatch is exactly what top-p addresses by sizing the shortlist dynamically.

Related: Top-p (nucleus) · Temperature · Greedy