Top, bottom, or both

The conflict

Two good rules point in opposite directions.

Caching wants instructions first. Prefix caching matches from the first token forward, so everything stable belongs at the top. Instructions are the most stable thing you have.

Attention favours the end. In a long window, an instruction 40,000 tokens above the question is competing with everything in between, and the model’s strongest attention is near where generation begins.

With a 2,000-token request the conflict does not exist; put the instructions at the top and stop thinking about it. With a 60,000-token request it is real, and teams discover it as “the model stopped following the format when we added the documents”, which is exactly what happened, and the format instruction did not move, the distance did.

The resolution most systems land on

A long stable block at the top and a short reminder at the bottom.

[system: role, policy, output contract, exemplars]     <- cached, ~1,200 tokens
[retrieved passages]                                    <- variable, ~9,000 tokens
[conversation history]                                  <- variable
[user question]
[reminder: the two constraints that matter most]        <- ~40 tokens, uncached

The reminder is not a copy of the system prompt. It is the one or two things that break when they are far away, which in practice are almost always the output format and the single most-violated constraint. Forty tokens, adjacent to the question.

You keep the cache discount on the large stable block, and you pay 40 uncached tokens for the part that needs proximity. That is the trade, and it is a good one.

Where the question goes

Separately from the instructions: with a long body of evidence, putting the question after the material is the common default, and it usually beats putting it only before.

The reasoning is causal attention. A question at the top is read before any evidence exists; the model cannot use it to guide reading, because there is nothing yet to read. A question at the bottom sits closest to the point of generation, in the strongest position available.

The stronger version is both: a short framing at the top so the model knows what it is reading for, and the full question at the bottom. This costs a few dozen tokens and is worth measuring, because the gain varies by model and by how long the evidence is.

What does not work

Making the instruction louder. Capital letters, “IMPORTANT”, “YOU MUST”, exclamation marks. These have a small effect at best, and they degrade as context grows, because the problem is distance rather than emphasis. Teams escalate the shouting instead of moving the text, and the prompt becomes unreadable to the humans who maintain it while behaving no better.

Repeating the whole prompt at the end. Doubles the instruction cost, and creates two copies that will eventually disagree when somebody edits one. The next lesson covers how to repeat safely.

Assuming the system role will carry it. The system role gets strong weighting at the top of a short request. In a 100,000-token context it is still just tokens, a very long way away.

Measure it the same way

The experiment is the same shape as the position probe. Fix everything, vary the instruction placement across three arms (top only, bottom only, top and bottom) and score adherence rather than answer quality, because format compliance is what you are testing and it is cheap to grade with a regex or a schema validator.

Expect small differences at short lengths and larger ones at long. Expect the “both” arm to win or tie, and to cost slightly more. Expect the result to differ between two models from the same vendor, which is the reason to re-run it when you upgrade.

The rule of thumb, stated as a rule of thumb

Below roughly 8,000 tokens, placement barely matters and caching should decide it. Above roughly 30,000 tokens, a bottom reminder is usually worth its tokens. Between the two, measure.

Those numbers are a starting point from common practice, not a finding. The point of the probe is that you can replace them with your own within a day, and then they are facts about your system rather than folklore about somebody else’s.