Context Windows and Long Context

The window as a budget: what long context really costs, the techniques that stretch it, and why the middle of a long prompt is a bad neighborhood.

Content last verified 2026-09.

Lessons

1. The Window Is a Budget
2. The Stretching Techniques
3. Lost in the Middle
4. Long-Context Economics

Sources

• Liu et al. (2023) — Lost in the Middle: How Language Models Use Long Contexts
• Xiao et al. (2023) — Efficient Streaming Language Models with Attention Sinks (StreamingLLM)
• Chen et al. (2023) — Extending Context Window of Large Language Models via Positional Interpolation
• Pope et al. (2022) — Efficiently Scaling Transformer Inference