Sliding Window Attention

In standard transformer attention, every token attends to every previous token · O(N²) memory and compute. Sliding window attention limits each token to the previous K tokens (window size). Enables long-context handling without quadratic blowup. See the related terms and live BenchGecko data for current examples.

Sliding window works because most useful context is local. Information that needs to travel far uses the "receptive field" expansion through stacked layers · a 32-layer model with 4K window has effective context of 128K through depth. Compute cost: sliding window attention is ~K/N cheaper than full attention when N >> K. See the related terms and live BenchGecko data for current examples.

Sliding window attention integrates with FlashAttention for memory-efficient implementation. The window size K is a hyperparameter: too small loses context, too large loses efficiency. Hybrid architectures (e.g., Gemma 2: sliding + global layers interleaved) balance quality vs speed. Mistral's effective context via sliding was questionable in practice · retrieval from far positions degraded. Modern approaches pair sliding with explicit long-context mechanisms (KV-cache compression, retrieval) to preserve quality.