The three formats, and the conversion trap

Three shapes you will meet

Instruction data comes in three formats. You will have to convert between them, and the conversion is where errors enter.

Alpaca style. Three fields, one of which is optional:

{"instruction": "Summarise the complaint in one sentence.",
 "input": "I ordered on the 3rd and it still hasn't...",
 "output": "Customer reports a delayed order placed on the 3rd."}

ShareGPT style. A conversation list with its own vocabulary:

{"conversations": [{"from": "human", "value": "..."},
                    {"from": "gpt",   "value": "..."}]}

Messages style. What the OpenAI API popularised and what has become the default:

{"messages": [{"role": "system", "content": "You are a triage assistant."},
              {"role": "user", "content": "..."},
              {"role": "assistant", "content": "..."}]}

Use messages style. TRL, Axolotl and Llama-Factory all accept it natively, it represents multi-turn and tool calls without extension, and it maps cleanly onto every model’s chat template. Convert everything to it on the way in, once, and never think about the others again.

The trap in Alpaca’s input field

Alpaca’s two-part prompt has to become one user message. Almost everybody concatenates them, and the question is how.

The original Alpaca template inserted specific headers: an instruction block, then an ### Input: block, then ### Response:. If you concatenate with a newline instead, you have changed the prompt format. That is fine, as long as you do it identically at inference time. It is a disaster if training used one separator and serving uses another, because the model learned to respond to a pattern that never arrives.

def to_messages(row):
    user = row["instruction"]
    if row.get("input"):
        user += "\n\n" + row["input"]        # decide once, apply everywhere
    return {"messages": [{"role": "user", "content": user},
                          {"role": "assistant", "content": row["output"]}]}

The rule that prevents the whole class of bug: the code that formats training examples and the code that formats inference requests must be the same function. Not similar. The same function, imported from one module, used by both.

The system prompt decision

Three options, and they have different consequences:

1. The same system prompt on every example, and at inference. The model learns to rely on it. Fine, and it is what most production systems do. Do not then change the system prompt after training without re-evaluating.
2. No system prompt anywhere. Also fine, and the simplest. The model’s behaviour is entirely in its weights.
3. Varied system prompts. Genuinely useful when you want the model to remain steerable: different tones, different output formats, selected at request time. Costs more data, because each variant needs examples.

The failure is mixing them by accident: half your examples carrying a system prompt because they came from a log that included one, half not. The model then learns that the system prompt is optional and roughly ignorable, which is the opposite of what you wanted.

Validate before you train

Four checks, worth the ten minutes:

for i, row in enumerate(data):
    m = row["messages"]
    assert m[-1]["role"] == "assistant", f"{i}: must end with assistant"
    assert all(x["content"].strip() for x in m), f"{i}: empty content"
    roles = [x["role"] for x in m if x["role"] != "system"]
    assert roles == ["user", "assistant"] * (len(roles)//2), f"{i}: bad alternation"
    assert len(tok.apply_chat_template(m)) <= MAX_LEN, f"{i}: too long"

The last one matters more than it looks. Examples over your maximum sequence length are silently truncated, and truncation removes the end of the assistant’s answer, including the end-of-turn token. Train on a few hundred of those and you have taught the model not to stop. Count how many examples exceed the limit before you train, and if it is more than a couple of percent, either raise the limit or drop those examples deliberately rather than letting the tokenizer decide.

The one-line test that catches most of it

Before any run, take one example, apply the chat template, and print the string:

print(tok.apply_chat_template(data[0]["messages"], tokenize=False))

Read it with your eyes. Every special token should be where you expect, the roles should be marked correctly, and there should be exactly one end-of-turn marker at the end. Five seconds, and it catches format bugs that would otherwise cost you a run and a day of confusion.