SHAP-Weighted Fusion Improves Multimodal Emotion and Sentiment Recognition

Multimodal emotion and sentiment recognition typically employs either early fusion, which concatenates different data modalities before classification, or late fusion, which combines predictions from independently trained unimodal models. Early fusion can be highly accurate but lacks transparency, while late fusion is modular but may miss crucial cross-modal interactions. This paper re-examines XAI-guided adaptive fusion (XGAF), a tree-based approach that blends unimodal and cross-modal experts, with sample-level weights derived from TreeSHAP attribution magnitudes. The study specifically investigated the impact of SHAP attribution reduction methods when experts have varying feature dimensionalities. It found that using sum-absolute reduction for SHAP attributions is critical, as mean-absolute and median-absolute reductions can suppress high-dimensional cross-modal experts. With sum-absolute reduction, XGAF achieved performance nearly matching or even slightly exceeding early fusion on benchmarks like MELD for 7-class emotion recognition and CMU-MOSEI for 3-class sentiment recognition. Ablation studies revealed that the primary performance gain comes from the inclusion of cross-modal experts, particularly the trimodal expert, rather than from complex per-sample routing. Diagnostics showed that sum-absolute weights concentrate on the trimodal expert, indicating its importance. This research provides a transparent empirical analysis of how SHAP reduction, expert dimensionality, and cross-modal expert design influence modular multimodal fusion, offering a more interpretable yet highly effective alternative to monolithic early fusion.