Rotary Position Embedding (RoPE) has become the de facto position encoding method in modern large language models, including LLaMA, GPT-4, and many others. This article traces the evolution of relative position encoding (RPE) from Shaw's additive approach through Transformer-XL's four-term reformulation, T5's bias-based method, and Swin Transformer's 2D extension, culminating in RoPE's elegant rotation-based solution. RoPE encodes position by rotating query and key vectors in attention, naturally capturing relative positions without additional parameters. Its key advantages include better length generalization, compatibility with linear attention, and seamless integration with existing architectures. Understanding this progression helps engineers make informed decisions about position encoding in new model designs. The analysis highlights trade-offs between expressiveness, computational efficiency, and implementation complexity across different RPE schemes.
A technical deep-dive into Rotary Position Embedding (RoPE) and its predecessors, explaining why it became the standard for LLMs.