Irreversibility Budget Proposed for Fleet-Level AI Agent Risk Control

As large language model (LLM) agents increasingly perform actions with irreversible consequences—such as financial transactions, code deployment, data deletion, or information disclosure—current security controls are proving insufficient. These controls typically check individual effects, allowing a fleet of agents, each individually authorized, to collectively exceed a principal's risk tolerance under a shared trigger, even if each local gate functions correctly. To address this, researchers propose the "irreversibility budget," a novel framework for fleet-level risk accounting and admission control. This budget functions as a cumulative account of residual value-at-risk, maintained by a trusted runtime for each principal across all agents, workflows, and tenants. Under this system, irreversibility is treated as a first-class resource. The runtime charges each agent action its residual loss, denying any marginal effect that would cause the aggregate risk to overdraw the budget. A controlled study demonstrated that while per-effect gates allowed fleet-level overdraws up to 48 times the tenant's risk limit, the irreversibility budget successfully kept all correctly charged runs within the specified limit. The primary challenge for deployment remains establishing accurate, dependency-aware pricing for heterogeneous and potentially adversarially declared effects.