Hugging Face, End to EndLesson 18 of 75

Contents

Hugging Face, End to End

Lesson 18 of 75. Sign in to keep your place.

Module 2 · Transformers, from characters to output

It loaded, it ran, the output is nonsense

9 min read

The order to check, not the order to guessInstruct?Is chat_templatesetTemplated?Not typed byhandSame repo?Tokenizer andweightsLeft padding?Only if batchedModel lastNine checksfirstEach step is under a minute and they are ordered by how often each is the cause. The model itself isthe tenth suspect, not the first, because fluent-but-wrong output is far more often a configurationmismatch than a bad checkpoint.
Each step is under a minute and they are ordered by how often each is the cause. The model itself is the tenth suspect, not the first, because fluent-but-wrong output is far more often a configuration mismatch than a bad checkpoint.

Figure

The worst category of bug

No traceback. No warning you noticed. The model produced fluent, confident, wrong text, or a classifier returned labels that look plausible and are random. Everything downstream works perfectly on garbage.

This lesson is a checklist in order of how often each cause is the cause. Work down it rather than guessing; each step is under a minute.

1. Is it an instruct model?

Python
print("chat_template:", tok.chat_template is not None)

False on a model you are prompting conversationally means you have a base checkpoint. A base model continues text; asked “What is the capital of Bihar?” it may reply with three more quiz questions. Nothing is broken. Get the -Instruct variant.

2. Did you apply the template?

Covered at length earlier, and it stays at number two because it stays that common. If you built the prompt with f-strings and role labels, stop and use apply_chat_template.

3. Is the tokenizer from the same repository as the weights?

Python
print(len(tok), model.config.vocab_size)

These should match, or differ only by a handful of added tokens. A serious mismatch means the integers mean different things to the two halves, which produces exactly the fluent-nonsense signature. This is the classic failure when loading an adapter against the wrong base.

4. Are the padding side and mask right?

If single calls are good and batched calls are bad, this is it. tok.padding_side = "left" for decoder-only generation, and pass the mask.

5. What dtype is it actually in?

Python
print(next(model.parameters()).dtype)

Two real failures here:

  • float16 overflow. Some models, notably the T5 family and a few older checkpoints, were trained in bfloat16 and overflow in float16: you get nan in the logits and empty or garbage output. bfloat16 has the same exponent range as float32 and does not; use it when the GPU supports it.
  • float16 on CPU. Slow, and on some builds numerically wrong. Use float32 on CPU unless you have measured otherwise.

A quick test: if torch.isnan(out.logits).any() is true, you have found it.

6. Is the quantization too aggressive?

4-bit is fine for chat and noticeably worse for arithmetic, code and long structured output. If a model is good at fp16 and bad at 4-bit on your task, that is the quantization, not a bug. Step up to 8-bit or a Q5_K_M GGUF and re-check before blaming anything else.

7. Are the generation parameters sane?

Python
print(model.generation_config)

Look for temperature above 1.2 with no top-p, a repetition_penalty above about 1.3 (which starts suppressing ordinary words), and do_sample: false ignoring everything you passed.

8. Did the input get truncated?

Python
print(enc["input_ids"].shape[-1], model.config.max_position_embeddings)

Silent truncation removes the end of a long document, usually the part that mattered. For a classifier reading complaints, this alone can halve accuracy.

9. For classifiers: what does id2label say?

{0: 'LABEL_0', 1: 'LABEL_1'} means nobody told the model which is which, and a coin flip on polarity is a 50% chance you have it backwards across your whole dataset.

10. Only now, suspect the model

After nine checks, it is reasonable to conclude the model is unsuited. Confirm it cheaply: run the same ten inputs through a well-known model of similar size. If that one is also poor, the task is harder than you thought or your inputs are unusual: both useful findings.

The minimal reproduction

When you report this to a discussion thread or a colleague, include: model id with revision, transformers and torch versions, the exact rendered prompt (tokenize=False), the generation kwargs, the dtype, and the device. Half the questions on the Hub cannot be answered because the rendered prompt is missing, and the rendered prompt is the answer about a third of the time.

Do this now

Deliberately break one working script four ways: drop the template, right-pad a batch, force float16 on a T5, and load the base instead of the instruct checkpoint. Look at each failure. You are building a library of signatures, and recognising a signature turns an afternoon into a minute.

Every lesson here is open. Your progress is recorded once you pass the Image Generation, in Practice exam.

Check your understanding

A T5-family summariser returns empty strings on a GPU under float16 but works on CPU under float32. What is happening?

Pick the one you would defend. Your answer stays private.

Next lesson · 11 minYour disk fills up before your patience does

Stuck on something?Ask Arjun, your AI tutor

Previous lesson: Showing tokens as they arrive, and making them stop

How to cite: Addaly (2026) ‘It loaded, it ran, the output is nonsense’, in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/loads-but-outputs-rubbish (Accessed: 13 October 2026).