New Benchmark Reveals LLM Agent Memory Limitations in Long Contexts

Large language models (LLMs) and LLM-based agents are increasingly used in various applications, from chat assistants to autonomous workflows. A critical factor for their reliability is "memory"—the ability to retain, access, and reason over information accumulated across long contexts and multiple interactions. Existing memory benchmarks often focus on simple fact retrieval or change detection. This paper introduces RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a new benchmark designed to rigorously evaluate compositional reasoning over long contexts. RECON comprises 24 case files, each 50k to 100k tokens long, spanning criminal, medical, and financial domains. It tests agents on six memory-intensive tasks, including reconstructing multi-hop evidence chains, propagating cascading invalidations, resolving source conflicts, and counterfactual reasoning. The evaluation of current architectures using RECON highlights substantial limitations. Even the strongest non-Oracle system achieved only 22.4% accuracy, indicating that both retrieval and reasoning capabilities remain significant challenges for LLM agents when dealing with extended and complex information. This benchmark provides a crucial tool for identifying and addressing these shortcomings in future agent development.