LLM Agents Struggle with Timely Abstention from Actions

This paper introduces and investigates the concept of "Agentic Abstention," which refers to an LLM agent's ability to recognize when a task is unachievable or ill-specified and to cease further actions. Unlike single-turn abstention, this is a sequential decision problem where agents must decide whether to answer, abstain, or gather more information over multiple turns. The study evaluated 13 LLM-as-agent systems and two agent scaffolds across over 28,000 tasks in web shopping, terminal environments, and question answering. The findings reveal a significant challenge: agents frequently struggle with timely abstention. Many either never abstain when appropriate or do so only after numerous unproductive interactions, particularly when the environment reveals a task's infeasibility after initial steps. Surprisingly, larger or more capable models sometimes performed worse at timely abstention. To address this, the researchers developed CONVOLVE, a context engineering method that distills interaction trajectories into reusable stopping rules. CONVOLVE substantially improved timely recall rates for Llama-3.3-70B on WebShop, demonstrating that effective abstention can be enhanced without model parameter updates.