Jet-Long Extends LLM Context with Dynamic Bifocal RoPE

Large Language Models (LLMs) are increasingly used in applications requiring very long contexts, often exceeding their pretraining window. Existing zero-shot context extension methods typically use a fixed rescaling factor, which either compromises short-context performance or fails at extremely long contexts. This research introduces Jet-Long, a novel tuning-free, zero-shot method designed to overcome these limitations. Jet-Long employs a dynamic bifocal RoPE (Rotary Position Embedding) strategy. It combines a local RoPE-faithful window with a long-range window whose rescaling factor adjusts dynamically based on the current sequence length. This design ensures that the base model's performance is recovered exactly for short inputs, while also allowing for clean extrapolation to significantly longer contexts. The method integrates an inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation, making the bifocal construction computationally efficient during inference. Fused into a single CuTe kernel, Jet-Long achieves up to 1.39x FA2 throughput on H100 GPUs during long-context prefill and incurs minimal overhead (<=4%) during single-batch generation across all lengths. Benchmarked on Qwen3-1.7B/4B/8B up to 128K context, Jet-Long consistently outperforms strong baselines, achieves the best overall accuracy on HELMET-RAG, and shows the lowest PG-19 perplexity. It also generalizes to hybrid attention architectures and demonstrates hyperparameter resilience, simplifying deployment.