Encoder Choice Critical for Multimodal Tabular-Image Learning.

A new study underscores the often-overlooked importance of encoder selection in multimodal learning, especially when combining tabular and image data. While plain Multi-Layer Perceptrons (MLPs) are commonly used for tabular data in such settings, the research argues that this approach may be suboptimal, given that tabular data remains a challenging domain for deep learning. The paper evaluates several state-of-the-art tabular models as encoders within an image-tabular framework, marking a novel exploration. A key challenge addressed was how to effectively embed In-Context Learning (ICL) models, which typically require labels for processing, to ensure consistent embedding of both training and test instances. The findings emphasize that a strong tabular encoder can significantly impact multimodal learning performance, suggesting that careful consideration of the tabular encoder is as vital as the image encoder for achieving optimal results.