Uncertainty-Aware RL Boosts Chemical Language Model Design Robustness

Reinforcement Learning (RL) is a powerful tool for designing new molecules using Chemical Language Models (CLMs), but current methods often overlook the inherent uncertainty in predicting molecular properties. This oversight can lead CLMs to explore highly uncertain regions of the chemical space, generating molecules with high predicted scores that lack strong support from training data, ultimately destabilizing the optimization process. To address this, researchers propose two complementary strategies for integrating predictive uncertainty into RL frameworks. The first approach treats uncertainty as an additional optimization objective, allowing the policy to balance exploitation of high-scoring molecules with the reliability of those predictions. The second method uses uncertainty to modulate policy updates, reducing the influence of molecules whose properties fall outside the confidence domain of the scoring function. These uncertainty-aware RL approaches were tested across various settings, including a controlled model system and real-world tasks using ChemProp models and Conformal Prediction. The results demonstrate that incorporating uncertainty enables CLMs to explore chemical space more robustly, favoring regions with lower uncertainty. This leads to more reliable discovery of desired molecules, significantly increasing the true hit rate without compromising the overall molecular score.