Examples that teach, and examples that mislead

Why a few examples beat a paragraph of rules

Describing a format in prose is lossy. Showing it is not. Three worked examples of the input-output pair you want will usually outperform two hundred words of description, because the model is very good at pattern continuation and only moderately good at following abstract instructions.

The effect is largest where the task is easy to demonstrate and awkward to define: tone, how much detail counts as enough, where to draw a category boundary, how to handle the messy input you actually receive rather than the clean one you imagined.

Classify each message as BILLING, TECHNICAL, ACCOUNT or OTHER.

Message: my card was charged twice this month
Label: BILLING

Message: app crashes when I open the reports tab
Label: TECHNICAL

Message: charged twice AND the app crashed while I was checking
Label: BILLING

Message: {{ input }}
Label:

The third example is the one earning its place. It resolves the ambiguity that would otherwise be resolved differently on every call: when a message spans two categories, billing wins. No amount of prose states that as clearly.

How many, and which

The returns curve is steep and then flat. Going from zero examples to three is usually a large improvement. Three to eight is a modest one. Beyond about ten you are mostly buying tokens.

Select for coverage of the boundary, not for typicality. If ninety per cent of your inputs are easy, examples of easy inputs teach almost nothing. Your examples should be the cases you would have to think about:

• The ambiguous case, with the tie-break shown
• The input that should produce the null or refusal output
• The malformed input: the empty message, the one word, the wrong language
• Two cases that look similar and get different labels

That last pair is the most valuable thing you can put in a prompt, because it defines a boundary by contrast rather than by description.

Three biases that bite

Order bias. Models are sensitive to the sequence of examples, particularly the last one. Put all your BILLING examples together at the end and you will tilt the classifier towards BILLING. Interleave labels, and if a class is rare, do not let it be missing entirely.

Label distribution bias. If eight of your ten examples are APPROVED, the model infers that approval is the common outcome and approves more. Balance the labels in your examples even when the real world is unbalanced, then handle the real prior in your own thresholds where you can see it.

Format lock-in, including the mistakes. The model copies everything, not just the part you meant. Trailing whitespace, inconsistent capitalisation, one example with a full stop and one without: all of it gets reproduced, and inconsistently, which is worse than either. Your examples should be byte-identical in structure.

Examples are training data you can edit

The practical loop that separates people who get good at this from people who do not:

1. Run your prompt over a hundred real inputs.
2. Find the ones it got wrong.
3. Pick two or three that represent distinct failure causes, correct them by hand, and add them as examples.
4. Re-run and check that nothing that was working broke.

Step four is not optional. Adding an example to fix one failure routinely breaks a different case, because you have shifted the pattern the model is matching. Without a fixed set to re-run, you are playing whack-a-mole with a blindfold on, and most teams do exactly that for months.

When the same edit keeps helping, you have discovered a rule. Promote it out of the examples and into the instructions, then delete the example. Instructions generalise; examples anchor. A prompt that grows only in examples becomes expensive and brittle.

The limitation

Examples teach format, boundary and tone. They do not teach facts, and they do not teach reasoning the model cannot already do.

Showing three examples of correct medical dosages does not make the fourth dosage correct: it makes it correctly formatted. This is the mistake behind a large fraction of disappointing few-shot results: someone adds examples hoping for accuracy and gets consistency instead. If the failure is the model does not know, the fix is retrieval or a tool, not another example. If the failure is the model knows but answers in the wrong shape, examples are exactly right.

One cost note. Examples sit in every request, so ten examples at 60 tokens each is 600 tokens on every call forever. Put them in the stable part of your prompt, before the variable material, so prompt caching can absorb them, which is the next lesson but one.