Logits, and the four knobs between them and words

What comes out of the model

out = model(**enc)
print(out.logits.shape)   # [batch, sequence_length, vocab_size]

For a causal model with a 151,936-token vocabulary and a 12-token prompt, that is a tensor of 12 × 151,936 raw numbers. Only the last row matters for generating the next token: it holds one score per possible next token, unnormalised, typically between about −20 and +20.

Softmax turns that row into a probability distribution. Then something has to choose, and everything people call creativity or randomness lives in that choice.

import torch
next_logits = out.logits[0, -1]
probs = torch.softmax(next_logits, dim=-1)
top = torch.topk(probs, 5)
for p, i in zip(top.values, top.indices):
    print(f"{p:.3f}  {tok.decode(i)!r}")

Run that on a half-finished sentence once. Seeing the actual candidates, and how often the top one holds 0.95 of the mass, makes every knob below intuitive rather than mystical.

Greedy, and why it repeats

Greedy decoding takes the argmax every step. It is deterministic, which is exactly what you want for classification-like tasks, extraction and anything you need to reproduce.

It also falls into loops. Once the model emits a phrase, that phrase raises the probability of itself recurring, and with no randomness nothing breaks the cycle, so you get ”the results show that the results show that the results show that”. This is a property of maximising at every step, not a defect in a particular model.

out = model.generate(**enc, do_sample=False, max_new_tokens=200)

Temperature

Temperature divides the logits before the softmax. Below 1 it sharpens the distribution: the likely tokens get likelier. Above 1 it flattens it. At 0 it is greedy.

• 0.0–0.3: extraction, classification, code, anything with a right answer.
• 0.7–0.8: general assistant use. Most chat defaults sit here.
• 1.0–1.3: brainstorming, variety, fiction.
• Above 1.5: usually incoherent, because low-probability tokens become reachable.

Top-k and top-p

Both truncate the candidate list before sampling.

• top-k keeps the k highest-probability tokens. Fixed count, so it keeps 50 candidates whether the model is certain or not.
• top-p (nucleus) keeps the smallest set whose probabilities sum to p. Adaptive: when the model is confident, that might be two tokens; when it is unsure, forty.

top-p is usually the better default precisely because it responds to the model’s confidence. top_p=0.9 with temperature=0.7 is the combination behind most defaults you will meet.

Repetition controls

repetition_penalty=1.1 divides the logits of tokens already present. no_repeat_ngram_size=3 forbids any 3-gram from occurring twice. The second is blunt: it makes a model literally unable to repeat a person’s name three times, which in a summary of a legal document is wrong. Prefer a mild repetition penalty and fix the rest with the prompt.

The interaction that wastes an afternoon

out = model.generate(**enc, temperature=0.9, top_p=0.95, max_new_tokens=100)

If the model’s generation_config.json has do_sample: false, this runs greedily and your temperature does nothing. Recent versions warn; older ones do not. The output is identical on every run and people conclude the model is broken or the seed is stuck.

Be explicit:

out = model.generate(**enc, do_sample=True, temperature=0.9, top_p=0.95,
                     max_new_tokens=100)

Inspect what you inherited with print(model.generation_config) before benchmarking anything. Two models that appear to differ in quality frequently differ only here.

Decoding back

text = tok.decode(out[0], skip_special_tokens=True)

generate returns the prompt plus the continuation, so trimming matters:

new_tokens = out[0][enc["input_ids"].shape[-1]:]
print(tok.decode(new_tokens, skip_special_tokens=True))

Without skip_special_tokens=True you see <|im_end|> and friends in your UI, which is the single most common cosmetic bug in a first demo.

Reproducibility

set_seed(0) makes sampling repeatable on the same hardware and library versions. Across GPUs, dtypes or versions, floating-point accumulation order differs and the same seed can produce different text. Determinism is per-configuration, not absolute, and any evaluation that assumes otherwise will chase ghosts.

Do this now

Generate the same prompt four times: greedy, then temperature=0.7, top_p=0.9, then temperature=1.5, then greedy with no_repeat_ngram_size=3. Four runs, one prompt. That comparison teaches more about decoding than any explanation.