Multimodal AI Improves Interpretable Clinical Predictions

Electronic health record (EHR) data inherently combines multiple types of information, or modalities. While integrating these modalities can boost predictive performance in clinical settings, many existing deep fusion methods obscure how individual data sources contribute to a prediction, hindering interpretability. Researchers propose an explicit multimodal routing framework designed for clinical prediction, focusing on interpretability, robustness, and auditability. This model processes three distinct EHR modalities: structured longitudinal variables, clinical notes, and chest X-rays. It constructs discrete unimodal, directional bimodal, and trimodal "routes" to capture both individual modality signals and the complex, asymmetric interactions between them. To enhance auditability and assess robustness, the framework includes inference-time route masking. This technique simulates missing modalities and reweights the remaining routes without requiring model retraining, allowing researchers to analyze changes in performance and routing weights. Evaluated on multi-label phenotype prediction and binary ICU mortality prediction using MIMIC-IV data, the framework revealed systematic differences in modality reliance across various clinical conditions, offering a transparent and practical approach to multimodal clinical AI.