Bayesian Reweighting Boosts Multimodal Retrieval Accuracy

Multimodal retrievers are crucial for knowledge-based visual question answering (VQA), as they retrieve external evidence to answer questions based on images. A common issue with current contrastive training methods is their tendency to treat all unmatched query-document pairs as equally uninformative negatives. This approach is problematic because many of these "unmatched" documents might still hold semantic relevance or partial usefulness, effectively being "false negatives" that mislead the training process. To address this, the researchers introduce Bayesian Data Reweighting, a novel probabilistic framework. This framework models the importance of query-document pairs as latent variables and adaptively infers posterior weights. These weights are then used to intelligently downweight likely false negatives during training. Through closed-form posterior updates under conjugate priors and stochastic EM optimization, the method consistently demonstrates improved retrieval accuracy. It has been successfully applied across three different retrievers and evaluated on seven knowledge-based VQA benchmarks, showing robust and consistent enhancements in performance.