MemTrace Benchmark Reveals LLM Long-Term Memory Failures

Large Language Model (LLM) agents are increasingly designed to maintain long-term memory of user-specific facts across multiple sessions. However, the conventional evaluation methods, which aggregate accuracy over individual questions or episodes, often mask critical limitations in how these models retain and utilize information. This approach treats each question independently, failing to reveal how a specific fact's understanding evolves or degrades under changing conditions. To address this, a new benchmark called MemTrace has been introduced. Unlike previous methods, MemTrace's fundamental unit of measurement is the "knowledge point," representing a single, typed fact about a user. This allows for a more granular analysis of memory behavior. MemTrace systematically probes each knowledge point across three controlled dimensions: the age of the memory (how many sessions ago the fact appeared), the type of question asked (current state, earlier state, or trajectory of change), and the evidence condition (whether evidence is present, missing, or contradicted by a false premise). Through evaluating 13 different memory-system configurations across four paradigms, MemTrace uncovered that seemingly similar overall accuracy scores often conceal distinct types of failures. For instance, an agent might successfully recall a fact's current and past states but fail to track how that fact has changed over time. Similarly, an agent might safely abstain from answering when uncertain but struggle to correct its understanding when presented with a false premise. A significant finding is that the primary bottleneck in long-term memory performance is not retrieval itself, but rather the effective use of reachable evidence; when systems failed, the necessary evidence was retrievable ten times more often than it was genuinely missing. These results suggest that improving LLM long-term memory requires a focus on better evidence utilization rather than just expanding storage or retrieval capabilities.