At every word, a language model has a probability for the ENTIRE vocabulary — tens of thousands of options, most of them nonsense. Top-k sampling keeps only the k most likely, throws the rest away, and re-normalises what’s left before rolling the dice. A small k is safe and boring; a large k is wild. Slide k and watch the shortlist shrink.
From the model’s softmax distribution over the vocabulary, top-k keeps the k highest-probability tokens, sets the rest to zero, and DIVIDES by their remaining mass so the survivors sum to 1 again — then samples from that. It is the crudest cousin of top-p (nucleus) sampling, which keeps a probability BUDGET rather than a fixed count. A fail-loud self-check throws unless exactly k tokens survive and the re-normalised distribution sums to 1.
A fixed k ignores the SHAPE of the distribution (when the model is confident, even k=5 includes junk; when unsure, it cuts good options) — which is why top-p adapts. The truncate-and-renormalise arithmetic is exact.