MedTVL AI Improves Medical Time Series Classification with Tri-Modal Synergy.

This paper presents MedTVL, an innovative AI architecture designed to enhance medical time series (MedTS) classification by integrating three distinct modalities: time series data, visual representations, and natural language. While existing methods often focus on bi-modal interactions, MedTVL explores the synergistic benefits of a tri-modal approach, drawing inspiration from how clinicians combine numerical assessments, visual inspections, and contextual information for diagnosis. The architecture features a convolution-based temporal pathway to capture fine-grained dynamics from raw numerical sequences and a transformer-based visual pathway to process holistic morphological structures derived from time-series images. Both pathways are adaptively guided by medical textual semantics to resolve diagnostic ambiguities. A Mixture-of-Experts mechanism dynamically routes each instance to specialized fusion experts, allowing for instance-specific reliance on temporal and visual outputs. MedTVL also supports multimodal contrastive learning to address the common challenge of clinical label scarcity. Extensive experiments across various medical datasets and tasks, including supervised, few-shot, and contrastive learning settings, demonstrate MedTVL's superior performance and transferability, highlighting its potential for robust clinical decision support.