OctoLong Enhances LLM Long-Context Code Understanding

The increasing demand for long-context capabilities in language models (LMs) for tasks like in-context learning and agentic workflows has highlighted a scarcity of suitable training data. Existing long-context corpora, often comprising books or academic articles, lack the dense, long-distance dependencies crucial for complex code understanding. To address this, researchers developed OctoLong, a novel context engineering pipeline. This pipeline utilizes an AST parser, language server backend, and package manager to recursively retrieve code references, enabling the creation of dependency-rich code contexts millions of tokens long. By mid-training a suite of open LMs, OctoLong-Instruct, on a mixture including these specialized code contexts, the models demonstrate substantial improvements in long-range retrieval, state tracking, and repository-level code understanding, significantly outperforming other state-of-the-art long-context LMs even when OctoLong data constitutes a small portion of the training mixture.