One big prompt, or four small ones

The mega-prompt and how it fails

A prompt that asks for six things at once (read this ticket, classify it, check it against policy, draft a reply, decide whether to escalate, and output JSON) fails in a characteristic way. It does not fail on all six. It degrades on two of them, usually the ones in the middle, and because the output still looks complete you do not notice for a month.

The mechanism is attention and instruction competition. Everything you asked for is competing for the same generation, and the parts that are hardest or least emphasised lose. You also cannot debug it: when the JSON has a wrong escalate flag, you have no way to know whether the classification was wrong, the policy lookup was wrong, or only the final decision was wrong.

Splitting is primarily a debuggability decision. Four prompts have four inspectable outputs, and each one can have its own test cases, its own model, and its own fix.

When to split

Split when any of these is true:

• Different steps deserve different models. Classification runs fine on the cheap tier; drafting the customer-facing reply may not. One prompt forces you to pay frontier prices for the classification too.
• A step’s output is checkable. If step two produces a category, you can validate it against a fixed list before step three ever runs. Errors caught early do not propagate.
• A step can be skipped. Half of tickets need no policy lookup. In a mega-prompt you pay for the lookup instructions every time.
• A step can be cached or reused. A document summary computed once serves twenty questions.
• You need to log an intermediate value for audit or for evaluation.

Keep it as one prompt when the steps are interdependent: where step two’s correct behaviour depends on nuance from step one that a short intermediate representation would destroy. Summarising then translating is worse than translating a summary written with translation in mind, because the intermediate throws information away.

The routing pattern

The most valuable chain in ordinary product work is two links: a cheap classifier that decides which specialised prompt handles the request.

kind = classify(msg)                    # small model, 40 tokens out, ~15ms
handler = {"billing": billing_prompt,
           "technical": tech_prompt,
           "account": account_prompt}.get(kind, generic_prompt)
return handler(msg, retrieve(kind, msg))

This wins three times over. Each specialised prompt is short, so it is cheaper and more compliant than one prompt trying to cover every case. Retrieval is scoped to the right corpus, which improves the hit rate more than any amount of prompt tuning. And when billing answers get worse, you edit the billing prompt without any risk to the other three.

The cost of the extra call is small: a 300-token classification on a cheap model is a fraction of a cent and 15 milliseconds. The cost of not routing shows up as a 1,500-token prompt on every request.

Errors compound, so put the checks between the links

The arithmetic is unforgiving. Four steps at 95 per cent each give 0.95⁴ ≈ 81 per cent end to end. Six steps at 90 per cent give 53 per cent. This is the same multiplication that makes agent demos fail, and a chain is a fixed agent.

The defence is that a fixed chain lets you put a deterministic check at each junction, and a check that can reject early stops the error from propagating:

cat = classify(msg)
if cat not in CATEGORIES:            # cheap, certain, catches drift
    cat = "other"
facts = extract(msg, cat)
validate(facts)                       # schema, ranges, required fields

Each check converts a silent wrong answer into a loud failure you can route or retry. That is the whole value: you cannot make each step more accurate by wishing, but you can stop a bad step from reaching the next one.

Latency, and what to run in parallel

A chain is serial by default, and four 800-millisecond calls is 3.2 seconds. Two structural fixes.

Fan out where steps are independent. Classification and PII redaction do not depend on each other; run them concurrently and pay for the slower one only.

Start the expensive step early. If retrieval does not depend on classification, fire it in parallel and discard the wrong branch. You paid for a query you did not need; you saved a round trip on every request.

And measure before you optimise. It is common for the whole chain to be dominated by one 2-second frontier call, in which case restructuring the cheap steps changes nothing a user can feel.

The failure mode of chains

One honest warning. A long chain becomes a system that nobody can hold in their head, where each step is individually reasonable and the whole produces answers no one can explain. Three or four steps is usually the point where the debuggability gain reverses.

If your diagram needs a diagram, you have not decomposed the problem; you have distributed it.