OOD-RL-Bench: New Framework for RL Out-of-Distribution Detection

Ensuring the reliability of reinforcement learning (RL) agents in dynamic and unpredictable environments requires robust detection of out-of-distribution (OOD) conditions. Current OOD detection benchmarks, however, are often designed for static datasets or image classifiers, failing to capture the complex, temporal, and action-dependent nature of RL trajectories. To bridge this gap, a new framework called OOD-RL-Bench has been developed. OOD-RL-Bench provides a comprehensive and extensible platform for evaluating OOD detectors by systematically injecting various types of anomalies into RL trajectories. The framework allows for easy integration of new detection methods and anomaly types. Initial evaluations using a Deep Q-Network in the LunarLander-v3 environment showed that while some anomalies, like observation perturbations and regime switches, are detectable with high accuracy, others, such as observation delay and action-conditioned dynamics, remain significantly challenging. The framework and results are publicly available for reproducibility.