LeRoPE Improves Language Models with Learnable Positional Encodings

Rotary Positional Encodings (RoPE) are a cornerstone of modern language models, responsible for encoding the relative positional information between tokens. Typically, the rotation rates within RoPE follow a fixed geometric sequence, controlled by a single hyperparameter. While previous work has explored adjusting this parameter or applying RoPE selectively, this research introduces a more fundamental enhancement. The proposed method, LeRoPE (Learnable RoPE), transforms these fixed rotation rates into learnable scalar parameters, allowing the model to dynamically optimize how positional information is encoded. The researchers trained a series of language models, ranging from 52 million to 2.5 billion parameters, to validate this approach. Across all scales, LeRoPE consistently outperformed standard RoPE and partial RoPE implementations. Notably, the largest models using RoPE required 3.4% more computational resources (FLOPs) to match LeRoPE's performance. The study also observed the emergence of a distinct, high-norm positional band within LeRoPE, indicating its unique learning behavior. This suggests that allowing the model to learn its positional frequencies leads to more efficient and effective language understanding.