Active Evaluation Framework Improves Robot Policy Testing Efficiency.

Evaluating generalist robot manipulation policies, especially those trained on vast and diverse datasets, presents a significant challenge due to the combinatorial explosion of task factors like object poses and camera viewpoints. Traditional exhaustive testing is impractical, and current narrow test suites often fail to uncover critical failure modes, leading to an inaccurate assessment of deployment readiness. To address this, researchers propose an active evaluation framework that re-frames policy evaluation as a sequential experimental design problem. This method employs a probabilistic surrogate model over a structured space of task factors. It then adaptively selects the most informative evaluation configurations, aiming to maximize information gain about the policy's performance distribution. Through 2,331 real-world evaluations across three tasks with varying factors, the framework demonstrated substantial efficiency gains. It typically reduced the number of required trials by 20-40% compared to conventional random testing, while effectively identifying failure-prone regions and characterizing policy behavior under diverse, previously unseen conditions.