New AI Model Improves Multimodal Emotion Recognition with Incomplete Data.

Researchers have introduced C$^2$MOE, a novel framework designed to overcome the challenges of incomplete data in multimodal emotion recognition within conversations. Traditional methods often struggle when modalities like audio or video are missing, leading to degraded performance. C$^2$MOE addresses this by unifying representation learning and missing modality imputation. The core of C$^2$MOE involves factorizing multimodal knowledge into consistency and complementarity components using interaction-aware experts. It maximizes cross-modal predictability for consistency and preserves modality complementarity by maximizing conditional entropy. A dual-branch prediction mechanism then imputes missing features, with a consistency branch minimizing uncertainty and a complementarity branch exploiting unique cues. A learnable reweighting module dynamically assigns importance, leading to robust and adaptive fusion.