AI Memory Evaluation Flawed by Presentation, Not Core Mechanism

AI systems are increasingly tasked with retrieving information from dynamic records that undergo revisions, such as issue threads or encyclopedic histories. The challenge extends beyond merely finding relevant data to discerning which claims remain valid, which have been superseded, and when to withhold information. While structured memories with typed edges and temporal updates promise solutions, evaluations often conflate changes in the underlying mechanism with changes in how information is presented to the AI. This study investigates this "Evidence-State Revision" problem by comparing different retrieval methods: flat retrieval, coarse edge invalidation, and a fine-grained RevisionLedger. They used 2,907 high-agreement questions from various sources like GitHub and Wikipedia. A crucial "render-matched control" experiment, where the layout was identical but deprecation features were disabled, exposed a central confound. The findings show that apparent gains from complex mechanisms like RevisionLedger are largely attributable to easier presentation. For instance, when a value was changed and then restored, RevisionLedger seemed to outperform a flat baseline by a significant margin, but most of this gain vanished when presentation was controlled, leaving a negligible residual from the fine-grained mechanism itself. The research concludes that memory evaluations must standardize presentation, and deprecation-aware systems should prioritize the coarsest retained state that satisfies their queries, as coarse invalidation proved more effective than fine-grained ledgers after controlling for presentation.