SpaR3D-MoE Enhances 3D Spatial Reasoning in MLLMs from Sparse Views

Multimodal Large Language Models (MLLMs) often struggle with integrating 2D semantic understanding with complex 3D spatial geometry, especially when relying on limited visual data. Current approaches either require extensive 3D-specific datasets or use basic RGB inputs with inefficient fusion methods, leading to issues like disrupted spatiotemporal connectivity and modality conflicts. To overcome these limitations, researchers have introduced SpaR3D-MoE, an end-to-end framework designed to equip MLLMs with advanced geometry-aware capabilities using only sparse RGB inputs. This system employs an adaptive spatiotemporal manifold sampling mechanism to create a geometry-aware graph, extracting crucial keyframes and maintaining scene topology while reducing redundancy. Furthermore, SpaR3D-MoE incorporates a heterogeneous geometry-inductive Mixture-of-Experts (MoE) architecture. An instruction-pose aware router directs multimodal tokens to specialized experts, effectively resolving the cross-modal contention common in monolithic fusion systems. This approach has demonstrated state-of-the-art performance across several benchmarks, significantly improving spatial reasoning tasks.