MassSpecGym Audit Reveals Critical Evaluation Flaws in AI Molecule Discovery

Reliable benchmarking is fundamental for advancing machine learning models in tandem mass spectrometry (MS/MS) based molecule discovery. However, subtle flaws in experimental design and evaluation procedures can undermine the trustworthiness of these benchmarks, leading to erroneous conclusions. This research conducts a comprehensive audit of model evaluation issues within the recent MS/MS machine learning literature, using the MassSpecGym benchmark suite as a primary case study. The review identified evaluation problems in a significant number of papers, specifically 17 out of 26 studies that reported MassSpecGym benchmark results within its first year of adoption. These issues were categorized into three main classes: data leakage, where training data inadvertently influences evaluation; shortcut learning, where models exploit unintended correlations rather than true underlying principles; and implementation bugs or metric divergences. Through extensive experimentation and code replication, the study quantifies the impact of these identified issues, demonstrating how they corrupt the evaluation standards that MassSpecGym was designed to enforce. The findings are distilled into generalizable recommendations for improving MS/MS challenges, benchmarks, and custom evaluation setups. To address these identified failure modes, MassSpecGym v1.5 has been released, incorporating the recommended corrections and publicly available for use.