MemOps Benchmarks LLM Memory Operations in Long Conversations.

Long-term memory is becoming essential for AI agents that engage in extended, multi-session conversations with users. However, existing benchmarks primarily evaluate this capability through downstream question-answering, which only scores the final correctness of an answer. This "black-box" approach fails to pinpoint the specific causes of memory failures, such as missing a fact's introduction, binding an operation to the wrong target, or using outdated information. The MemOps benchmark addresses this by reframing conversational memory as a dynamic lifecycle of explicit operations, including remembering, forgetting, updating, and reflecting. It represents each memory event with a structured trace that details its trigger, target, scope, state transition, and supporting evidence. A controllable generation pipeline embeds these operations into long, task-oriented conversations and produces gold operation traces along with six categories of operation-level probes. Initial evaluations using MemOps reveal distinct failure modes that final-answer accuracy alone obscures, showing that current systems are not uniformly reliable. For instance, session-level retrieval performs better than turn-level, and long-context models struggle with reconstructing ordered memory-state trajectories.