The diagnostic ladder for a bad output

The order matters because the cheap causes are the common ones

When a local model produces something poor, there are six possible causes and people almost always investigate them in the wrong order: starting with the model and ending with the template, when the reverse is correct.

Work down this list. Each step is minutes, and most problems are resolved in the first three.

1. Is it actually running on the hardware you think?

nvidia-smi during generation. If utilisation is zero, the model is on the CPU and everything else you observe is a symptom of that. Thirty seconds, and it explains a surprising share of “the model is bad” reports where the real complaint was speed.

2. Is the prompt what you think it is?

Print the exact string the model receives, template and all.

print(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))

Look for: a missing or wrong template, a doubled beginning-of-sequence token, a system prompt that has silently vanished because the model has no system role, and a missing generation prompt. Then check the token count against your configured window. If the count exceeds the window, the model never saw the beginning of your input and nothing further is worth investigating.

3. Is generation stopping in the right place?

If the output runs on, invents the next user turn, or trails into unrelated text, the end-of-turn token is not in the stop list. Add it explicitly and re-test.

4. Are the sampler settings sane?

Set temperature to 0 and turn everything else off. If the output becomes good, the problem was in the sampler and you can reintroduce settings one at a time. If it stays bad, sampling was never the issue and you have eliminated a whole area.

This single test (everything off, temperature 0) is the highest-value diagnostic in the list, because it cleanly separates two halves of the problem space.

5. Is it the quantisation?

Run the same prompt against a higher-bit quant of the same model, or against a hosted copy of the full-precision weights. If the higher precision is clearly better, you have found it. If not, the quant is exonerated and you have a real answer rather than a suspicion.

This is worth doing before any hardware decision, because “buy more memory to run a better quant” is only justified once you have evidence the quant is the constraint.

6. Is it the prompt design?

Too many simultaneous instructions, key constraints buried in the middle, format described rather than demonstrated, no examples. Rewrite along the lines of the prompting lessons and re-test on the same ten items.

7. Is it the model?

Only now. Run the same prompts against the same weights hosted at full precision. If they fail there too, the weights cannot do the task, and the options are a different model, a different size class, decomposition into smaller steps, or retrieval so the answer comes from a document instead of the model’s memory.

Symptom to cause, at a glance

• Fluent but ignores the system prompt: template or system-role support.
• Repeats a phrase until the token limit: sampler; try DRY.
• Answers well then invents a conversation: missing stop token.
• Forgets the start of a long document: context window, silently truncated.
• Produces prose where JSON was requested: no grammar, no anchoring.
• Good on short inputs, confidently wrong on long ones: past your working context limit.
• Broken indentation and missing brackets: repetition penalty too high.
• Correct but unbearably slow: not a quality problem at all; go back to step 1.
• Different answers to the same prompt at temperature 0: floating-point and batch composition, not a bug.

The habit that makes this work

Change one thing at a time and keep the same ten test items in front of you throughout. Two changes at once produce a result you cannot attribute, and a new set of test items produces a comparison you cannot make.

And write down what fixed it. Every one of these problems recurs (on the next model, on the next machine, when a colleague hits the same wall) and a five-line note is the difference between remembering the answer and rediscovering it.