What examples cost, and what they buy

Examples are the most expensive instruction format

A well-chosen example teaches something a paragraph cannot: the exact shape of an output, a boundary case, a tone. It also costs 150 to 400 tokens, on every request, forever.

So the question is never “do examples help” (they usually do) but “do these four examples buy more than 1,000 tokens of something else would”. That is a measurable question and almost nobody measures it.

The curve you should expect

Going from zero to one example is usually a large improvement, especially for format. One to three is a smaller improvement. Three to eight is often nothing, and past that the returns are close to flat for most tasks while the cost keeps climbing linearly.

So the working default is two to four, chosen to cover the cases that differ, not the cases that are typical. Four examples of the same easy shape teach one thing four times; one typical, one boundary case, one where the answer is a refusal, and one in a second language teach four.

The refusal example is the one most often missing. If your system must sometimes say “the passages do not answer this”, show it doing so. A model that has seen only successful answers is measurably less willing to produce that output, and the gap costs you exactly on the questions where a wrong answer is most damaging.

Three biases examples introduce

These are properties of how models read examples, and they are stable enough to design around.

Recency. The last example has disproportionate influence on the output’s shape. If your examples vary in length or tone, the final one sets the expectation. Put the most representative example last, deliberately.

Label distribution. If four of your five classification examples are labelled approved, the model over-predicts approved, regardless of the input. Balance the labels across the examples even when the real distribution is skewed, or state the real prior explicitly in words.

Format lock-in. Models copy surface features of examples very readily: a leading capital, the absence of a full stop, a particular date format. This is useful when you want it and a nuisance when an example contains an accidental quirk. Read your examples as text with this in mind; a stray trailing space in an example can become a stray trailing space in every answer.

Static or dynamic

Static examples are the same on every call, sit in the cached prefix, and cost almost nothing once the cache is warm.

Dynamic examples are retrieved per request: the three most similar solved cases from your archive. Quality is usually better, sometimes much better, because the examples are about the same kind of thing as the question.

The cost is not the retrieval. It is the cache. Dynamic examples placed near the top of the request change the prefix on every call, so everything below them becomes a cache miss. On a system with a large stable prefix, that can be a larger bill than the examples themselves.

Two ways to keep both:

1. Put the dynamic examples below the stable block, after the cached prefix ends. You keep the discount on the system prompt and pay full price for the examples.
2. Bucket them. Pick from a fixed set of five example bundles by request type, so there are five possible prefixes rather than one per request. Five cache entries is fine; a million is not.

Examples and untrusted text

One rule with no exceptions: examples are instructions, so they belong to the trusted part of the window and they must never be built from unreviewed user content.

Building a dynamic example set out of “similar past conversations” pulls arbitrary user text into the most authoritative position in your prompt. Somebody’s message, once retrieved as an exemplar, is read as a demonstration of correct behaviour. That is an injection path with a queue attached.

If you build examples from real cases, they get reviewed by a person, redacted, and stored as fixed assets, not retrieved live from a conversation table.

The experiment worth running once a quarter

Drop from four examples to two and run your evaluation set. Then to zero, with the instruction strengthened to describe what the examples were showing.

A surprising number of systems lose nothing. Examples accumulated during early development, when the instruction was vague and the model was worse, and nobody rechecked after either improved. Recovering 800 tokens on every request is a real saving, and the only cost of finding out is one afternoon and one evaluation run.

Where the examples do earn their place, you will see it clearly in the numbers, and you will now be able to say by how much, which is a better answer than “we’ve always had them”.