DreamBench-SWE Benchmarks Software Agent Memory Hygiene

DreamBench-SWE is a specialized benchmark created to rigorously test the memory management capabilities, or "memory hygiene," of software agents across multiple sessions. This benchmark is crucial because many real-world software tasks require agents to recall and utilize information from previous interactions that cannot be inferred from the current context alone. The tasks are scored using executable hidden oracles, ensuring objective evaluation. The research reports on two folds of the benchmark, v2 and v2.1, which involved hundreds of work units and cells across various memory conditions. The findings from the successor audit, v2.1, demonstrated that while some hosted memory configurations showed better performance, no external memory system achieved high pass rates. Deterministic verbatim event memory and a typed-plus-raw reference probe performed moderately better. The audit confirms DreamBench-SWE's effectiveness as a discriminating benchmark for profiling executable agent memory. However, it also highlights that establishing a clear superiority among external memory mechanisms or broad product generality remains challenging, indicating that effective long-term memory for software agents is still an active area of development.