New Method Discovers Autonomous System Failures Efficiently

This research introduces an adaptive method for efficiently discovering failures in autonomous systems, particularly when testing budgets are limited. Autonomous systems often exhibit rare and diverse failure modes, making comprehensive real-world testing challenging. While proxy systems like simulators or lower-fidelity models can be extensively sampled, their failures frequently do not translate accurately to real-world target systems due to "sim-to-real" or "system-to-system" gaps. The proposed method addresses this by effectively leveraging information from proxy systems alongside a small number of target system evaluations. It learns a local predictor of target risk by correcting proxy failure signals using a residual modeling approach inspired by control variates. To ensure both likelihood and diversity in discovered failures, this predictor is combined with a support-aware mutual-information objective. This objective prioritizes realistic and well-supported failure regions while simultaneously expanding coverage across various failure modes. Evaluations across diverse tasks, including autonomous driving, manipulation, and quadruped velocity-tracking, demonstrated the method's effectiveness. It discovered up to twice as many failures compared to random sampling and active-learning baselines. Crucially, it identified severe and diverse failures that competing methods missed, highlighting its potential for more robust system validation.