ContextWeave Benchmark Evaluates LLM Agent Memory in Workflows

As language agents move beyond isolated tasks to handle long-horizon, stateful workflows, effective memory management becomes paramount. However, existing evaluations often oversimplify memory to mere retrieval or question answering, failing to capture its impact on real-world, multi-step processes. To address this, researchers introduced ContextWeave, a novel longitudinal benchmark designed to evaluate how an agent's recalled experience influences its performance in realistic office-work streams. The benchmark reconstructs privacy-preserved, multi-month workflows from 14 participants into over a thousand executable tasks, complete with instructions, containerized environments, and task-specific rubrics. Findings indicate that actionable, experience-rich memory significantly improves workflow continuation and reduces redundant exploration, outperforming compact summaries. While recall generally boosts performance across various base models, it also highlights the susceptibility to misleading information, underscoring the need for memory systems that optimize both retrieval relevance and reliable execution.