Cross-Modal Integration Boosts Audio Sentiment Analysis

Automatically discerning sentiment from speech is a complex task that requires analyzing both vocal inflections and the semantic content of spoken words. While recent audio foundation models have made progress, their ability to fully capture all relevant aspects remains a question. To address this, researchers propose a novel multimodal approach that combines audio and textual information. This solution leverages automatically generated text transcripts, which are further translated into multiple languages using machine translation tools. These diverse textual modalities are then integrated with audio features through a cascaded architecture of cross-modal transformer blocks. A key aspect of this work involves knowledge distillation: the insights gained from the comprehensive multimodal "teacher" model are transferred to a more efficient, audio-only "student" model. Experiments on a large dataset confirm that automatically generated multilingual textual information significantly improves sentiment polarity classification, and the distillation process enhances the audio-only model's performance without adding computational burden during inference.