New Benchmark Reveals AI Agents Over-Refuse Safe Actions

As Large Language Model agents take on more autonomous roles in workplaces, their ability to make critical "pre-commit" decisions—whether to proceed with an action or hold for human review—becomes paramount. SteerBench-Work is a new incident-anchored, bidirectional benchmark designed to evaluate these steering decisions across various enterprise domains like developer operations, customer service, and finance. The benchmark includes 106 real-world scenarios and their evidence-reversed mirrors, with an even split between "proceed" and "hold" labels to fairly assess both types of errors. The initial findings from testing 30 model conditions reveal a significant imbalance in error types: models wrongly hold authorized, evidence-cleared work in 28.1% of cases, but wrongly allow unsafe work in only 1.0%. This indicates a strong bias towards caution, or "over-refusal." Interestingly, higher-capability models often exacerbate this issue, demonstrating that general intelligence does not equate to proper steering calibration. The benchmark also highlights that models perform significantly worse on evidence-reversed versions of famous incidents compared to the incidents themselves, suggesting a lack of robust understanding of risk resolution.