Multimodal EHR Foundation Models Improve Clinical Prediction

Autoregressive foundation models trained on electronic health records (EHRs) show promise for zero-shot clinical prediction, but most currently rely solely on structured event codes. This research introduces a framework to integrate multiple clinical modalities, such as ECG waveforms, chest X-ray images, and clinical notes, into these models in a structured way. The framework utilizes modality-specific latent compression and gated cross-attention with temporal alignment. Key design choices investigated include how to compress long per-modality sequences to reduce computational overhead and improve generalization, and the impact of different pretrained encoders for each modality. Experiments on the MIMIC-IV dataset revealed that optimized latent compression configurations significantly outperform uncompressed or simple pooling methods. Stronger pretrained encoders consistently yielded better performance. However, merely adding auxiliary modalities does not guarantee improvement; careful fusion architecture design and appropriate clinical evaluation are essential for achieving better outcomes in tasks like ICU mortality prediction.