Agent-Safety Benchmarks Lack Consistency, Often Confuse Safety with Capability

A recent validity audit critically examines four prominent agent-safety benchmarks (R-Judge, InjecAgent, AgentHarm, AgentDojo), revealing that their scores are often quoted interchangeably despite measuring distinct behaviors. The study highlights fundamental problems with the metrics used, noting that simple "always positive" policies can achieve deceptively high F1 scores on some benchmarks, outperforming models that actually discriminate. The audit found that the three broad-coverage benchmarks rank the same models inconsistently, and the observed disagreements are often artifacts of small panel sizes. Crucially, the research indicates a negative correlation between model capability and misalignment safety on certain panels, suggesting that more capable models might not necessarily be safer in terms of alignment. While AgentHarm showed the strongest association with jailbreak safety, the authors caution that this indicates convergent validity for harmful compliance rather than general safety. The study concludes that any safety claim requires explicit naming of the benchmark, metric, target behavior, and model panel to be meaningful, emphasizing that current benchmarks often confuse safety with mere capability.