The quadratic complexity of standard attention mechanisms has long been a bottleneck for scaling transformer models to long sequences. This survey explores the key innovations that address this challenge: sparse attention patterns that limit computation to relevant tokens, linear attention that approximates the attention matrix, state space models (SSMs) that offer recurrent alternatives, and hybrid architectures that combine these approaches. Each method offers distinct trade-offs in terms of accuracy, speed, and memory usage. For example, sparse attention excels in tasks with local dependencies, while SSMs provide strong performance on long-range sequences. Understanding these architectures is essential for AI engineers optimizing models for production, as they enable longer context windows, lower latency, and reduced hardware costs. This analysis provides a roadmap for selecting the right architecture based on task requirements and computational constraints.
A survey of attention architectures that overcome O(L²) complexity, covering sparse, linear, SSM, and hybrid methods.