New Framework Boosts VLM Physical Reasoning Without Training

Video-language models (VLMs) excel at video understanding but often falter when it comes to reasoning about physical plausibility, struggling with object interactions, causal dynamics, and fundamental physics principles. This limitation is evident in their performance on complex physical reasoning benchmarks. To address this, a new framework called PhysMRV has been developed.PhysMRV operates without requiring additional training or parameter updates for the VLM. It transforms existing training videos into a "Hierarchical Memory Bank" containing structured physical knowledge. This bank comprises three levels: scene descriptions for visual context, physical-event graphs for object interactions and causality, and physics-rule summaries for general principles.During inference, PhysMRV retrieves relevant physical memories from this bank. It then leverages the structured evidence within these memories to guide a frozen VLM in verifying the physical plausibility of observed events. Evaluations on benchmarks like ImplausiBench, IntPhys2, and GRASP Level 2 show consistent and significant improvements over direct prompting across various state-of-the-art VLMs, demonstrating the effectiveness of structured physical memories in enhancing reasoning.