{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"attention-sinks-and-the-first-tokens","title":"Why the first tokens are special","url":"https://addaly.com/learn/context-engineering/attention-sinks-and-the-first-tokens","position":39,"of":87,"minutes":8,"answer":"Softmax forces attention to sum to one, so heads with nothing to attend to dump mass onto the earliest tokens, making them structural attention sinks.","keyIdea":"Softmax forces attention to sum to one, so heads with nothing to attend to dump mass onto the earliest tokens, making them structural attention sinks, which is why evicting the first tokens from a KV cache destroys generation, and why the start of a request should hold your most stable, most authoritative text."},"body":{"format":"markdown","text":"## A mechanism, not a superstition\n\nAttention scores are passed through a softmax, which forces them to sum to one across all positions. That constraint has an odd consequence: a head that has nothing it particularly wants to attend to still has to put its attention somewhere.\n\nWhat models learn to do, consistently, is dump it on the earliest tokens. Those positions become **attention sinks**: visible in attention maps as a bright column at the start, receiving a large share of the mass from many heads at many layers, regardless of what they contain. The first token is often literally a beginning-of-sequence marker with no semantic content at all, which is the clue that this is a mechanism rather than a meaning.\n\nThe finding was made concrete in the StreamingLLM work in 2023, and it explains something that had puzzled people building long-running chat systems.\n\n## The bug it explains\n\nIf you are serving a model yourself and want unbounded conversation, the obvious approach is a sliding window over the key-value cache: keep the most recent N tokens, evict the rest, generate forever in constant memory.\n\nIt fails badly. Output quality collapses (not degrades, collapses into repetition and nonsense) as soon as the window slides past the very beginning of the sequence.\n\nThe reason is the sinks. Evicting the first few tokens removes the destination for a large share of attention mass, and the distribution rearranges itself onto tokens that were never meant to carry it. The fix in the paper is almost comically small: keep the first four tokens permanently, slide the window over everything else, and coherent generation continues over millions of tokens.\n\nIf you run your own inference with a custom cache policy, this is a load-bearing detail. On a hosted API it is somebody else's problem, already solved.\n\n## What it means for assembly\n\nThree things follow for anyone building context, whoever runs the model.\n\n**The primacy half of the U curve has a mechanism.** Early positions are structurally privileged, not merely conventional. That is a reason to expect the effect to persist across model generations rather than to be trained away.\n\n**The very start of your request is the most valuable real estate you have.** It should hold the most stable, most authoritative content you own: your role and policy, not a timestamp, not a session id, and certainly not user-supplied text. This aligns neatly with what caching wants, which is a rare case of two constraints agreeing.\n\n**Any operation that alters the beginning of the sequence is more disruptive than it looks.** Rotating a system prompt for an experiment, injecting a per-request greeting, or prepending a variable header changes the tokens the sinks sit on, in addition to breaking the cache.\n\n## What it does not mean\n\nThree over-readings to avoid.\n\nIt does not mean the first tokens are read more carefully. Sink attention is largely content-independent: the mass goes there because it has to go somewhere, not because the token is important. Putting your critical instruction in position 0 does not make it obeyed by this mechanism.\n\nIt does not explain the whole U. Recency comes from causal attention and from where generation begins; training-data structure contributes as well. Sinks are one factor among several.\n\nAnd it does not give you a knob. There is no way, through an API, to change how a model allocates sink attention. This lesson is here so you can recognise a class of failure and stop attributing it to the wrong cause, not because there is a setting to adjust.\n\n## Where it becomes practical\n\nThree places, all of them for people running their own inference with `llama.cpp`, vLLM, or a bespoke loop:\n\n- **Cache eviction policies.** Keep the first few tokens. Every serious implementation now does; verify yours.\n- **Context compaction.** When you rewrite a conversation into a summary, you are rebuilding the sequence from the start, including its sinks. This is one reason compaction produces a sharper behavioural discontinuity than the size change alone would suggest.\n- **Quantisation and cache compression.** Some methods degrade the first tokens more than others, and the resulting quality loss is disproportionate. If you are compressing a KV cache, treat the sink positions as special.\n\nFor everyone else, the takeaway is smaller and still useful: the beginning of your request matters structurally, it is the one place where caching and attention want the same thing, and it should be the most boring, most stable text in your system."},"check":{"question":"A team implements a sliding-window KV cache for unbounded chat. Output degenerates into repetition once the window passes the start of the conversation. What is the fix implied by the mechanism?","options":[{"text":"Increase the window size, since the degeneration comes from having too little recent context.","correct":false,"tempting":"A larger window only delays the failure; it recurs whenever the window eventually slides past the beginning."},{"text":"Retain the first few tokens permanently and slide the window over the rest.","correct":true},{"text":"Lower the temperature, since repetition indicates a sampling problem.","correct":false,"tempting":"Repetition can come from sampling, but here it appears exactly when the cache policy changes, which points at the cache."},{"text":"Re-encode the whole conversation periodically so positions are recomputed from zero.","correct":false,"tempting":"Would work by accident by restoring the sinks, at the cost of recomputing everything: the expensive version of keeping four tokens."}]},"prev":{"slug":"few-shot-examples-inside-a-budget","title":"What examples cost, and what they buy","url":"https://addaly.com/learn/context-engineering/few-shot-examples-inside-a-budget"},"next":{"slug":"what-a-token-costs","title":"The arithmetic of a request","url":"https://addaly.com/learn/context-engineering/what-a-token-costs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why the first tokens are special', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/attention-sinks-and-the-first-tokens (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why the first tokens are special","courseTitle":"Context Engineering","module":{"index":4,"slug":"position-and-order","title":"Position, order and what the model actually reads"},"answer":null,"keyIdea":"Softmax forces attention to sum to one, so heads with nothing to attend to dump mass onto the earliest tokens, making them structural attention sinks, which is why evicting the first tokens from a KV cache destroys generation, and why the start of a request should hold your most stable, most authoritative text.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-mechanism-not-a-superstition","runs":[{"text":"A mechanism, not a superstition"}]},{"t":"p","runs":[{"text":"Attention scores are passed through a softmax, which forces them to sum to one across all positions. That constraint has an odd consequence: a head that has nothing it particularly wants to attend to still has to put its attention somewhere."}]},{"t":"p","runs":[{"text":"What models learn to do, consistently, is dump it on the earliest tokens. Those positions become "},{"text":"attention sinks","b":true},{"text":": visible in attention maps as a bright column at the start, receiving a large share of the mass from many heads at many layers, regardless of what they contain. The first token is often literally a beginning-of-sequence marker with no semantic content at all, which is the clue that this is a mechanism rather than a meaning."}]},{"t":"p","runs":[{"text":"The finding was made concrete in the StreamingLLM work in 2023, and it explains something that had puzzled people building long-running chat systems."}]},{"t":"h2","id":"the-bug-it-explains","runs":[{"text":"The bug it explains"}]},{"t":"p","runs":[{"text":"If you are serving a model yourself and want unbounded conversation, the obvious approach is a sliding window over the key-value cache: keep the most recent N tokens, evict the rest, generate forever in constant memory."}]},{"t":"p","runs":[{"text":"It fails badly. Output quality collapses (not degrades, collapses into repetition and nonsense) as soon as the window slides past the very beginning of the sequence."}]},{"t":"p","runs":[{"text":"The reason is the sinks. Evicting the first few tokens removes the destination for a large share of attention mass, and the distribution rearranges itself onto tokens that were never meant to carry it. The fix in the paper is almost comically small: keep the first four tokens permanently, slide the window over everything else, and coherent generation continues over millions of tokens."}]},{"t":"p","runs":[{"text":"If you run your own inference with a custom cache policy, this is a load-bearing detail. On a hosted API it is somebody else’s problem, already solved."}]},{"t":"h2","id":"what-it-means-for-assembly","runs":[{"text":"What it means for assembly"}]},{"t":"p","runs":[{"text":"Three things follow for anyone building context, whoever runs the model."}]},{"t":"p","runs":[{"text":"The primacy half of the U curve has a mechanism.","b":true},{"text":" Early positions are structurally privileged, not merely conventional. That is a reason to expect the effect to persist across model generations rather than to be trained away."}]},{"t":"p","runs":[{"text":"The very start of your request is the most valuable real estate you have.","b":true},{"text":" It should hold the most stable, most authoritative content you own: your role and policy, not a timestamp, not a session id, and certainly not user-supplied text. This aligns neatly with what caching wants, which is a rare case of two constraints agreeing."}]},{"t":"p","runs":[{"text":"Any operation that alters the beginning of the sequence is more disruptive than it looks.","b":true},{"text":" Rotating a system prompt for an experiment, injecting a per-request greeting, or prepending a variable header changes the tokens the sinks sit on, in addition to breaking the cache."}]},{"t":"h2","id":"what-it-does-not-mean","runs":[{"text":"What it does not mean"}]},{"t":"p","runs":[{"text":"Three over-readings to avoid."}]},{"t":"p","runs":[{"text":"It does not mean the first tokens are read more carefully. Sink attention is largely content-independent: the mass goes there because it has to go somewhere, not because the token is important. Putting your critical instruction in position 0 does not make it obeyed by this mechanism."}]},{"t":"p","runs":[{"text":"It does not explain the whole U. Recency comes from causal attention and from where generation begins; training-data structure contributes as well. Sinks are one factor among several."}]},{"t":"p","runs":[{"text":"And it does not give you a knob. There is no way, through an API, to change how a model allocates sink attention. This lesson is here so you can recognise a class of failure and stop attributing it to the wrong cause, not because there is a setting to adjust."}]},{"t":"h2","id":"where-it-becomes-practical","runs":[{"text":"Where it becomes practical"}]},{"t":"p","runs":[{"text":"Three places, all of them for people running their own inference with "},{"text":"llama.cpp","code":true},{"text":", vLLM, or a bespoke loop:"}]},{"t":"ul","items":[[{"text":"Cache eviction policies.","b":true},{"text":" Keep the first few tokens. Every serious implementation now does; verify yours."}],[{"text":"Context compaction.","b":true},{"text":" When you rewrite a conversation into a summary, you are rebuilding the sequence from the start, including its sinks. This is one reason compaction produces a sharper behavioural discontinuity than the size change alone would suggest."}],[{"text":"Quantisation and cache compression.","b":true},{"text":" Some methods degrade the first tokens more than others, and the resulting quality loss is disproportionate. If you are compressing a KV cache, treat the sink positions as special."}]]},{"t":"p","runs":[{"text":"For everyone else, the takeaway is smaller and still useful: the beginning of your request matters structurally, it is the one place where caching and attention want the same thing, and it should be the most boring, most stable text in your system."}]}],"check":{"question":[{"text":"A team implements a sliding-window KV cache for unbounded chat. Output degenerates into repetition once the window passes the start of the conversation. What is the fix implied by the mechanism?"}],"options":[{"text":[{"text":"Increase the window size, since the degeneration comes from having too little recent context."}],"correct":false,"tempting":[{"text":"A larger window only delays the failure; it recurs whenever the window eventually slides past the beginning."}]},{"text":[{"text":"Retain the first few tokens permanently and slide the window over the rest."}],"correct":true,"tempting":null},{"text":[{"text":"Lower the temperature, since repetition indicates a sampling problem."}],"correct":false,"tempting":[{"text":"Repetition can come from sampling, but here it appears exactly when the cache policy changes, which points at the cache."}]},{"text":[{"text":"Re-encode the whole conversation periodically so positions are recomputed from zero."}],"correct":false,"tempting":[{"text":"Would work by accident by restoring the sinks, at the cost of recomputing everything: the expensive version of keeping four tokens."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"few-shot-examples-inside-a-budget","title":"What examples cost, and what they buy","minutes":9},"next":{"slug":"what-a-token-costs","title":"The arithmetic of a request","minutes":10}}}