The Stretching Techniques
Lesson 2 of 4 in Context Windows and Long Context.
Why is a window the size it is? Because the model was trained at that length. Position schemes — learned tables, RoPE rotations — only behave inside the range the model saw during training, so the window is a trained capability, not a config flag you can raise. Genuine Context extension therefore means changing the position scheme (position interpolation, frequency-aware RoPE scaling) and then training the model to live with the change — Chen et al. (2023) showed the interpolation half; the mechanics live in Positional Encoding and we won’t repeat them here.
What this lesson adds is the serving-time family that gets marketed under the same “longer context” banner but does something different. Two goals travel under one name:
- Extending relational reach — making the model genuinely able to attend from token 90,000 back to token 12. That is what position-scheme extension plus fine-tuning buys.
- Managing serving memory and stability — keeping the KV cache bounded and generation stable as input keeps arriving. That is what sliding windows and attention sinks buy.
The honest distinction: the second family does not let the model see more. It decides what the model stops seeing, gracefully.
| Technique | What it extends | What it does not |
|---|---|---|
Position interpolation / scaled RoPE + fine-tune | True relational reach: after fine-tuning, Attention operates in-distribution across the whole new window | The bill. A genuinely longer window still pays full prefill compute and full KV memory for every token — and quality at the new lengths must be measured, not assumed (next lesson) |
Sliding-window attention (trained-in architecture) | Affordability: each layer attends within a fixed span, so per-layer attention cost and cache growth stop scaling with total length | Direct reach. No single layer can relate two tokens farther apart than the window; longer-range influence must hop through the layer stack indirectly. This is a design baked in at training time — imposing a window on a model trained with full attention degrades it |
Attention sinks + rolling cache (StreamingLLM) | Stable generation length: with the first few tokens pinned and a recent window rolling, a model can keep generating on streams far beyond its trained length with a bounded cache (Xiao et al., 2023) | Recall. Evicted tokens are gone — the model cannot answer questions about content that left the cache. This is memory management, not memory |
The StreamingLLM result deserves its one paragraph, because the observation behind it is genuinely strange. Xiao et al. (2023) noticed that autoregressive LLMs park a large share of attention on the first few tokens of the sequence — almost regardless of what those tokens say. They called them attention sinks. The practical consequence: the naive way to bound a cache — keep only the most recent N tokens and evict the oldest — collapses the model’s output quality the moment those first tokens fall out. Keep just a handful of initial tokens plus the recent rolling window, and the paper reports stable language modeling on streams vastly longer than the training window, with no fine-tuning.
Read the claim precisely, because vendors sometimes don’t: this makes infinite-length generation feasible with finite memory. It does not make an infinite context — a question about the middle of yesterday’s stream fails, because those tokens were evicted. When a product says “unlimited context”, ask which of the two families it means.
Key terms: Context window, Context extension, Sliding-window attention, Attention sink, KV cache
Why the first tokens become sinks
The explanation Xiao et al. offer starts from an inconvenient property of Softmax: attention weights at every position must sum to exactly 1. A query that finds nothing relevant in the context still has to spend its full attention budget somewhere. The natural dumping ground is a token that is visible to every later position under Causal masking — and the only tokens visible to all positions are the earliest ones. Over training, the model learns to use the initial positions as a parking lot for unneeded attention mass, largely independent of their content (the paper shows the effect persists even when the first token is meaningless).
That reframes the eviction failure: dropping the initial tokens does not remove information the model needs — it removes the resting place the attention distribution was trained to rely on, so every query’s weights renormalize over the wrong support and the output distribution degrades sharply. Hence the two fixes in the paper: always keep a few initial tokens in the cache alongside the rolling window, or train the model from the start with a dedicated learnable sink token so future serving only has to pin one. This is also why Attention sink behavior matters beyond streaming — cache-eviction and cache-compression schemes in general have to treat “rarely attended” and “safe to evict” as different claims.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.