Why the first tokens are special

A mechanism, not a superstition

Attention scores are passed through a softmax, which forces them to sum to one across all positions. That constraint has an odd consequence: a head that has nothing it particularly wants to attend to still has to put its attention somewhere.

What models learn to do, consistently, is dump it on the earliest tokens. Those positions become attention sinks: visible in attention maps as a bright column at the start, receiving a large share of the mass from many heads at many layers, regardless of what they contain. The first token is often literally a beginning-of-sequence marker with no semantic content at all, which is the clue that this is a mechanism rather than a meaning.

The finding was made concrete in the StreamingLLM work in 2023, and it explains something that had puzzled people building long-running chat systems.

The bug it explains

If you are serving a model yourself and want unbounded conversation, the obvious approach is a sliding window over the key-value cache: keep the most recent N tokens, evict the rest, generate forever in constant memory.

It fails badly. Output quality collapses (not degrades, collapses into repetition and nonsense) as soon as the window slides past the very beginning of the sequence.

The reason is the sinks. Evicting the first few tokens removes the destination for a large share of attention mass, and the distribution rearranges itself onto tokens that were never meant to carry it. The fix in the paper is almost comically small: keep the first four tokens permanently, slide the window over everything else, and coherent generation continues over millions of tokens.

If you run your own inference with a custom cache policy, this is a load-bearing detail. On a hosted API it is somebody else’s problem, already solved.

What it means for assembly

Three things follow for anyone building context, whoever runs the model.

The primacy half of the U curve has a mechanism. Early positions are structurally privileged, not merely conventional. That is a reason to expect the effect to persist across model generations rather than to be trained away.

The very start of your request is the most valuable real estate you have. It should hold the most stable, most authoritative content you own: your role and policy, not a timestamp, not a session id, and certainly not user-supplied text. This aligns neatly with what caching wants, which is a rare case of two constraints agreeing.

Any operation that alters the beginning of the sequence is more disruptive than it looks. Rotating a system prompt for an experiment, injecting a per-request greeting, or prepending a variable header changes the tokens the sinks sit on, in addition to breaking the cache.

What it does not mean

Three over-readings to avoid.

It does not mean the first tokens are read more carefully. Sink attention is largely content-independent: the mass goes there because it has to go somewhere, not because the token is important. Putting your critical instruction in position 0 does not make it obeyed by this mechanism.

It does not explain the whole U. Recency comes from causal attention and from where generation begins; training-data structure contributes as well. Sinks are one factor among several.

And it does not give you a knob. There is no way, through an API, to change how a model allocates sink attention. This lesson is here so you can recognise a class of failure and stop attributing it to the wrong cause, not because there is a setting to adjust.

Where it becomes practical

Three places, all of them for people running their own inference with llama.cpp, vLLM, or a bespoke loop:

• Cache eviction policies. Keep the first few tokens. Every serious implementation now does; verify yours.
• Context compaction. When you rewrite a conversation into a summary, you are rebuilding the sequence from the start, including its sinks. This is one reason compaction produces a sharper behavioural discontinuity than the size change alone would suggest.
• Quantisation and cache compression. Some methods degrade the first tokens more than others, and the resulting quality loss is disproportionate. If you are compressing a KV cache, treat the sink positions as special.

For everyone else, the takeaway is smaller and still useful: the beginning of your request matters structurally, it is the one place where caching and attention want the same thing, and it should be the most boring, most stable text in your system.