LLMZero Discovers Adaptive RL Training Strategies Using LLM Agents.

Reinforcement Learning (RL) post-training strategies are highly dependent on the specific dataset being used. A recurring pattern observed is that model capacity parameters tend to increase steadily across training stages, whereas regularization parameters frequently fluctuate in response to dynamic training conditions. This distinction is crucial because fixed training schedules cannot adapt to the non-stationary exploration-exploitation trade-offs that regularization must manage. This principle provides actionable guidelines for designing multi-stage training processes. Researchers discovered this insight through LLMZero, a novel system that employs large language model (LLM) agents to explore various training trajectories. These agents utilize a tree search mechanism to diagnose issues at each checkpoint and then propose coordinated adjustments to multiple parameters. Across four diverse GRPO tasks, LLMZero successfully identified strategies that improved upon baseline models by 9% to 140% relatively, and outperformed grid search by 6% to 15% relatively. It consistently surpassed both random search and skill-based agents. The underlying structural principle, which dictates how parameters evolve, proved transferable across different tasks, explaining why the discovered strategies, despite their qualitative differences, exhibit similar parameter dynamics.