Padding on the wrong side ruins generation quietly

Why padding exists at all

A batch is a rectangle. Three sentences of 4, 11 and 6 tokens cannot be stacked into a tensor until they are the same length, so the short ones get filled with a pad token:

enc = tok(["hi", "a much longer sentence here"], padding=True, return_tensors="pt")
print(enc["input_ids"])
print(enc["attention_mask"])

The mask is the important half. It is 1 for real tokens and 0 for padding, and the attention mechanism uses it to make padded positions contribute nothing. Pass input_ids without the mask and the model attends to the padding as if it were content, a mistake that is easy to make when people build tensors by hand and then wonder why batched results differ from one-at-a-time results.

Rule: whatever the tokenizer returns, pass all of it. model(**enc), never model(enc["input_ids"]).

The side matters, and only for generation

For an encoder model doing classification, padding on the right is fine. Every position is visible to every other, the mask zeroes the pad, the pooled output is correct.

For a decoder-only model doing generation, right padding is a silent quality bug. Generation continues from the last position of the sequence. If the sequence ends in padding, the model is being asked to continue from a pad token, having seen your prompt some distance back. Depending on the model and the mask handling, you get output that is degraded, generic, or occasionally empty. No error, no warning, and the same prompt run alone produces a good answer, so people conclude that batching breaks the model.

The fix is one line:

tok.padding_side = "left"

Now every sequence ends with its real final token and generation starts from the right place. Do this for every AutoModelForCausalLM you batch. For training a causal model, right padding is correct, because labels are shifted and the loss ignores padded positions. The side depends on the operation, not on the model.

The missing pad token

Many decoder-only checkpoints ship with no pad token, because generation one at a time never needs one. Batch them and you get:

ValueError: Asking to pad but the tokenizer does not have a padding token.

The standard fix:

if tok.pad_token is None:
    tok.pad_token = tok.eos_token

This is fine for inference. For training it is worth thinking about once: with pad_token == eos_token, a naive label setup can teach the model that end-of-sequence is followed by more end-of-sequence, which makes it fail to stop. The usual answer is to set padded label positions to -100, which the loss ignores, and the data collators in the library do this for you.

Truncation, and the side of that too

enc = tok(texts, padding=True, truncation=True, max_length=1024, return_tensors="pt")

padding=True pads to the longest item in the batch. padding="max_length" pads everything to max_length, which wastes compute but produces uniform shapes: occasionally required by compiled or exported models.

truncation_side defaults to right. For a chat history, cutting from the right removes the most recent turn, which is the one that mattered. Set tok.truncation_side = "left" when the end of the input is the important part.

The cost of padding

Padding is wasted computation. A batch of 32 where one item is 2,000 tokens and the rest are 50 runs as if all 32 were 2,000: roughly a fortyfold waste. Two cheap mitigations:

1. Sort by length before batching, so each batch is internally uniform. This alone often doubles throughput on mixed-length data and costs about four lines.
2. Cap the batch by token count rather than by item count (for example 8,000 tokens per batch) so a batch of long items is small and a batch of short items is large.

Serving stacks like vLLM and text-generation-inference remove the problem entirely with continuous batching, but the sorting trick works today in a plain script and needs no new dependency.

The five-line sanity check

After any change to batching, run one input alone and the same input inside a batch of eight, and compare:

solo = pipe("What is the capital of Karnataka?")
batched = pipe(["filler"] * 7 + ["What is the capital of Karnataka?"])[-1]

They should agree. When they do not, the padding side is the first suspect, the missing mask the second.

Do this now

Take any small instruct model, batch four prompts of very different lengths with the default right padding, and look at the shortest prompt’s answer. Then set padding_side = "left" and run the identical batch. The difference is the bug this lesson exists for, and seeing it once makes it permanent.