Domain Adaptation Boosts Molecular Language Models for Discovery

Pretrained molecular language models (MLMs) are increasingly used as encoders to understand structure-property relationships in molecules. However, their effectiveness for molecular discovery, especially when moving beyond their initial pretraining domain, has been an open question. This research systematically benchmarks several MLMs across diverse virtual molecular libraries relevant to drug discovery, organic materials, and catalysis. The study found that the performance of native MLM embeddings varied substantially across different libraries, and surprisingly, traditional molecular fingerprints often provided a more consistently robust baseline. This variability suggested a potential mismatch between the MLM's pretraining domain and the target discovery domain. Crucially, the research shows that explicit domain adaptation significantly enhances representation performance. Fine-tuning MLM encoders using structures from the specific target virtual library consistently improved sample efficiency. In many benchmark tasks, these adapted encoders emerged as the top-performing representations, demonstrating that the quality of molecular representations is highly dependent on the target domain and that tailored adaptation is a promising strategy for efficient adaptive decision-making in virtual screening and self-driving laboratories.