Vera Framework Automates LLM Agent Safety Testing at Scale

Large language model (LLM) agents are increasingly capable of performing autonomous actions using external tools, which introduces complex and rapidly evolving safety risks. Current safety testing methods often target expert-designed violations and rely on hard-coded rules, making them difficult to scale and adapt as agents evolve. To overcome these limitations, researchers have developed Vera, an automated, end-to-end safety testing framework that applies software engineering principles to non-deterministic agents. Vera operates through a self-reinforcing, three-stage pipeline. First, it continuously discovers and structures emerging risks into taxonomies of safety risks, attack methods, and tool execution environments, drawing from existing literature. Second, it generates executable safety cases by combinatorially composing elements from these taxonomies, each specifying a safety goal, an initial state, and a deterministic verification predicate based on observable artifacts. Third, Vera adaptively executes heterogeneous agents in isolated sandboxes, where a control agent manages multi-turn interactions, and evidence-grounded verifiers judge outcomes based on environment state and tool-call evidence, rather than the model's self-reported behavior. Evaluations on four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code) exposed substantial safety weaknesses, with average attack success rates reaching 93.9% under multi-channel attacks. The release of Vera-Bench, comprising 1600 executable safety cases, underscores the necessity of modular, executable testing infrastructure for rigorous and maintainable safety evaluation of rapidly evolving agentic systems.