LLMs Unify Multimodal Clinical Prediction with Textual Serialization

This paper introduces a simplified, unified approach for multimodal clinical prediction using large language models (LLMs). Instead of relying on complex, task-specific fusion architectures that combine dedicated encoders for each data modality (like free-text narratives, vital signs, and lab values), the proposed method converts all patient data into a single, coherent natural language sequence. This serialized text is then used to fine-tune a pre-trained language model end-to-end, without any architectural modifications for fusion. The effectiveness of this approach was evaluated across three distinct clinical prediction tasks: in-hospital mortality, graft failure prediction, and emergency triage classification. The results consistently showed that this unified textual serialization method matched or even surpassed the performance of established task-specific multimodal baselines. Notably, it also outperformed a gradient boosting model currently used in clinical practice for graft failure prediction. These findings suggest that a single, serialization-based paradigm can significantly reduce system complexity while maintaining or improving predictive accuracy in multimodal clinical settings.