New Method Boosts Multimodal Sentiment Analysis with Unseen Modalities.

A new challenge in multimodal sentiment analysis, termed Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), has been identified. Traditional approaches often struggle when faced with modality combinations during testing that were not present in the training data, leading to poor generalization. This problem is particularly relevant in real-world scenarios where data incompleteness and variability are common. To tackle IMSAUMC, a new model called Contrastive Mixed Prompt Learning (CMPL) has been proposed. CMPL incorporates a label-guided contrastive feature learning mechanism to develop robust and discriminative cross-modal representations. It also utilizes modality-combination prompts with a soft router to better learn diverse combinations. Furthermore, CMPL introduces three prompt contrastive learning strategies, which are crucial for enabling the model to effectively learn prompts corresponding to unseen modality combinations, thereby significantly enhancing its generalization capabilities. Extensive experiments on three widely used datasets demonstrate that CMPL achieves over a 5% improvement in accuracy compared to state-of-the-art methods.