New Benchmark Evaluates LLM Agent Abstention Capabilities

As large language model agents are increasingly deployed for autonomous tasks, their ability to recognize when *not* to act is a critical safety concern that has largely gone unevaluated. This research addresses this gap by presenting AgentAbstain, a novel framework and benchmark specifically designed to test agentic abstention. The framework employs a paired-task design, where each task has a "should-act" version and a corresponding "should-abstain" variant, created through controlled perturbations to instructions, tools, or environment states. AgentAbstain comprises 263 paired tasks across 42 executable sandbox environments, generated by an automated pipeline called AbstainGen to ensure scalability and resist data contamination. Evaluations across 17 frontier LLMs and 4 agent harnesses reveal that even the best agent (Gemini 3.1 Pro) achieves only 59.5% paired accuracy. Crucially, abstention capability appears independent of general task-solving ability, suggesting that simply scaling up LLMs will not resolve this safety issue. The study also identifies common failure modes, such as agents performing irreversible actions before recognizing the need to abstain.