RL Researchers Must Differentiate Simulator Use Cases

A new position paper highlights a critical distinction that reinforcement learning (RL) researchers need to make regarding their use of simulators. The authors argue that there are two fundamentally different objectives: either "solving simulators" to achieve peak performance within the simulated environment, or "using simulators as a proxy" to develop agents that can learn effectively in real-world deployment settings. The paper explains that these two use cases impose different constraints on how an agent interacts with the simulator, dictate which algorithms are appropriate, and require distinct evaluation metrics. For instance, solutions optimized purely for simulator performance might not translate well to real-world scenarios where an agent learns while deployed. The authors provide examples and simple experiments to illustrate how blurring this distinction can lead to misleading conclusions and practices. They advocate for the RL community to clearly articulate their simulator usage, fostering more relevant and impactful research for both simulator-specific challenges and real-world learning applications.