Active-GRPO Boosts Molecular Optimization with Adaptive Learning

Training large language models for scientific reasoning, particularly in instruction-based molecular optimization, faces challenges like the collapse of multi-step reasoning with supervised fine-tuning and sparse feedback in reinforcement learning. Reference-guided Policy Optimization (RGPO) methods attempt to mitigate this by using dataset-provided references, but their effectiveness is capped by the quality of these references. This research proposes Active-GRPO, an active reasoning paradigm designed to overcome this limitation. Active-GRPO allows the policy to adaptively decide, on an instance-by-instance basis, whether to imitate a reference or to reinforce its own novel discoveries. This is achieved through two coupled mechanisms: active imitate-reinforce and active referencing. The former performs imitation learning when references are superior, then shifts to self-improvement via reinforcement learning once the policy generates better candidates. The latter continuously upgrades the reference itself with the best policy-generated molecules, ensuring that guidance remains informative and progressively raises the performance target throughout training. This approach significantly improves molecular optimization metrics, demonstrating substantial gains over existing methods like GRPO and RePO.