PROPEL Boosts Task Generation for Reinforcement Learning Agent Training

Training AI agents through reinforcement learning often hits a wall due to a lack of appropriate training tasks. These tasks need to be challenging enough to advance the model but still solvable. Current methods for generating such tasks often produce trivial, impossible, or poorly defined scenarios, especially as AI models become more capable. The PROPEL framework tackles this by enabling the training of task generators. These generators are optimized to produce tasks that meet specific validity and learnability criteria. A key innovation is a lightweight "activation probe" that predicts a task's solve rate without needing to run the full, time-consuming solver, making the generator training process much more efficient. Experiments across various domains, including mathematics, code generation, and software engineering, demonstrate PROPEL's effectiveness. It significantly increases the proportion of generated tasks that fall within the desired learnable frontier, leading to more efficient and effective training for advanced AI models.