PredicateLongBench Reveals LLM Long-Context Limitations

While large language models (LLMs) have shown rapid improvements in handling long contexts, existing benchmarks often measure only average performance and can be easily saturated. There's a clear need for a more systematic way to evaluate how models perform as task difficulty scales. To address this, a new benchmark called PredicateLongBench has been developed. PredicateLongBench specifically stress-tests long-context reasoning by requiring models to identify the longest contiguous subsequence of words in a lengthy input that satisfies certain predicates or constraints, such as lexicographic ordering. The core innovation lies in its ability to systematically explore multiple axes of difficulty, probing various aspects of long-context understanding. The benchmark offers two generation pipelines: one fully synthetic using random word-like strings, and another using real-world documents while preserving their statistical properties. Evaluations using PredicateLongBench demonstrate that even frontier models struggle considerably as task difficulty increases along these defined axes, underscoring the current limitations in their long-context capabilities. The tasks are conceptually simple, avoiding the need for LLM-based generations or judges, which enhances robustness.