Examples, and how to choose them

Why examples help small models more

A large instruction-tuned model has strong priors about what you probably want. A small one does not, so showing it three worked cases moves it further. On classification and extraction tasks, going from zero examples to three or five is often a bigger improvement than doubling the model size, and it costs a few hundred tokens.

What the examples actually convey is worth understanding, because it changes how you choose them. Research on in-context learning found that corrupting the labels in the examples (pairing inputs with wrong answers) degraded performance far less than expected, while removing the examples entirely, or changing their format, hurt a lot. The examples are mostly teaching the model three things: what the input space looks like, what the label space is, and exactly what shape the output should take. The correct mapping is carried less than intuition suggests.

That does not mean you should use wrong labels. It means the highest-return properties of an example set are format consistency and coverage of the label space, not subtlety of the individual answers.

How many

Three to eight for most tasks. The curve flattens quickly, and every example is context you pay for on every request. Twenty examples in the prompt is usually a sign that the task should be fine-tuned instead: a later block covers exactly that trade.

Six rules for choosing them

Cover every label at least once. A model shown four billing examples and one fault example will over-predict billing. This majority-label bias is strong at small sizes and is the most common defect in hand-written example sets.

Include the hard cases, not the clear ones. Examples that a rule could have handled teach nothing. The example worth its tokens is the ambiguous one where you show which way to resolve it.

Keep the format byte-identical across examples. Same field order, same separators, same capitalisation. Any inconsistency is a licence for the model to be inconsistent too.

Watch the ordering. The last example has the most influence. Do not put all of one label at the end, and if you rotate examples, keep the order fixed within a run so results are comparable.

Use real inputs. Invented examples are cleaner, better punctuated and more polite than what users actually send, and a model tuned on them handles real messiness worse.

Include one example of the boundary case. What to do when the input is empty, off-topic, or unanswerable. Without it the model has never seen a refusal in your format and will not produce one.

Selecting examples per request

For a task with diverse inputs, a fixed set of five examples is a compromise. Retrieving examples similar to the current input usually does better:

# index once
emb = model.encode([e["input"] for e in pool], normalize_embeddings=True)

# per request: take the 4 nearest labelled examples
sims = emb @ model.encode([user_input], normalize_embeddings=True).T
picked = [pool[i] for i in sims.ravel().argsort()[-4:]]

This is a small retrieval system in front of your prompt and it is often worth several points on a heterogeneous task. Two cautions: keep the label distribution of what you retrieve balanced, or a query that resembles many billing tickets pulls four billing examples and biases the answer; and put the selected examples in a fixed order so runs stay comparable.

Measuring instead of guessing

Example choice is easy to test and rarely tested. On your fifteen development items:

1. Zero examples. Record the score.
2. Three fixed examples covering all labels. Record.
3. Eight fixed examples. Record.
4. Four retrieved examples. Record.

Half an hour, four numbers, and you will usually find that the difference between three and eight is smaller than the difference between a balanced set and an unbalanced one. That result is more useful than any general advice, including this lesson’s.

The one-line summary a colleague can act on: examples set the format and the label space, so make them consistent, balanced and real, and spend your attention on coverage rather than on cleverness.