ReMem Boosts MLLM Long Video Understanding with Adaptive Memory

A novel framework named ReMem has been introduced to significantly improve Multimodal Large Language Models (MLLMs) in understanding long videos without requiring additional training. MLLMs typically struggle with long video contexts due to limited context windows, often relying on keyframe selection methods that can overlook crucial temporal information. ReMem addresses this by implementing a temporal granularity-adaptive keyframe selection framework with dual-level memory augmentation. At the query level, a Memory-Driven Question Parsing component uses LLM long-term memory to decode question temporal granularity and extract semantic entities. At the video level, Synergistic Dual-Semantic Frame Alignment leverages intrinsic structural memory to align frames with query semantics, guiding Structure-Aware Dynamic Frame Routing to optimally distribute sampling budgets. By explicitly preserving temporal information through these memory mechanisms, ReMem effectively suppresses redundancy and empowers MLLMs to perform robust multi-granular video reasoning. Evaluations across four popular LongVideoQA benchmarks, using three MLLMs, demonstrated highly efficient, state-of-the-art zero-shot performance, with notable gains on LVBench and LongVideoBench.