Cross-Modal Representation Alignment Improves Time-to-Event Prediction in Healthcare.

Predicting time-to-event (TTE) outcomes from diverse clinical data, such as CT imaging and electronic health records (EHR), presents significant challenges due to imbalances and distribution shifts between modalities. This study proposes a novel foundation model-driven framework designed to align cross-modal representations, aiming for improved generalization across various clinical tasks and institutions. The framework independently encodes CT and EHR data using domain-specific foundation models, then aligns them in a shared latent space through four distinct fusion strategies: late fusion, contrastive alignment, cross-attention, and co-attention. The researchers evaluated the framework on two distinct TTE tasks: predicting pulmonary embolism (PE) mortality and cardiovascular disease (CVD) outcomes, using large-scale multi-institutional cohorts. Consistent improvements in concordance index (1.5-5.4%) were observed over unimodal baselines when both modalities contributed comparably. Specifically, contrastive multimodal fusion, particularly with CLMBR representations, demonstrated the most consistent and statistically robust enhancements, especially for PE mortality. For major adverse cardiovascular events (MACE), cross-attention achieved the highest internal performance, while image-guided co-attention excelled in external performance. These findings underscore that a "one-size-fits-all" approach to fusion is ineffective, establishing task-aware multimodal alignment as a critical design principle for achieving robust generalization and scalable clinical deployment.