MedMix Enhances Federated AI for Multimodal Medical Data

Federated multimodal medical AI faces significant challenges due to modality heterogeneity, which occurs both at the client level (where clients may lack access to certain modality types) and at the sample level (where individual patient records might have incomplete modality subsets). Sparse Mixture-of-Experts (MoE) architectures show promise for adapting computation to different modalities, but their application in federated learning becomes fragile when modality configurations vary across clients. This fragility can lead to local routing policies diverging and experts developing incompatible specializations. To overcome these issues, researchers propose MedMix, a semantic-alignment framework specifically for federated multimodal sparse MoEs. MedMix coordinates cross-client routing and expert specialization by leveraging modality context. On the client side, it employs modality-context-aware routing to guide expert selection based on each token's modality identity, position, and incompleteness. Across clients, it uses consensus-guided routing alignment to establish server-side consensus anchors for common modality patterns, thereby aligning local routing distributions. Furthermore, MedMix incorporates client-adaptive expert aggregation, which uses client-specific modality-pattern prototypes to match and aggregate functionally similar experts across different clients. Experiments on real-world multimodal medical datasets demonstrate that MedMix achieves the best average F1 score across various settings of modality heterogeneity and incompleteness, showing particularly strong gains in scenarios with severe heterogeneity.