Offline Dynamics Models Aid Real-World RL Hyperparameter Selection

Deploying reinforcement learning (RL) in real-world systems faces a significant hurdle in hyperparameter selection, especially when live experimentation is expensive or simulators are unavailable. Previous research proposed using calibration models, trained on offline data, to mimic environment dynamics and facilitate offline hyperparameter tuning. However, these methods had only been tested in simplified simulated environments. This paper presents the first real-world application of such calibration models within an industrial context: a municipal water treatment plant. The researchers evaluated several calibration model approaches, including a k-nearest neighbors model with a Laplacian distance metric, using high-dimensional, non-stationary sensor data for next-step prediction tasks. The findings demonstrate that these models can generate realistic long-horizon rollouts and accurately capture hyperparameter sensitivity trends. The study also explored how calibration models scale with year-long datasets, support fine-tuning learning rates for pre-trained agents, and maintain robustness under distribution shifts. This work provides a crucial proof of concept for using offline dynamics models to support RL deployment in complex, real-world settings, while also highlighting areas for future research.