Agent Evaluation Requires Outcome Finality and Cross-Unit Separation.

Evaluating AI agents accurately requires more than just observing their state at the end of a run. This research highlights two critical conditions often overlooked: "outcome finality" and "cross-unit separation." Outcome finality ensures that the scored result won't change due to delayed operations, while cross-unit separation guarantees that one agent's run doesn't inadvertently affect the state or outcome of subsequent runs. Without these, reported scores may not reflect the true performance. The study demonstrates that delayed operations can indeed alter terminal labels, and persistent service states between runs can impact subsequent scores. A review of ten public evaluation protocols revealed that while they define when a run stops and what is scored, they are less consistent in documenting unfinished operations or providing evidence for treating runs as truly separate. To address this, the paper proposes an "open-effects record" to track any lingering operations or resources that could influence outcomes or subsequent trials, thereby improving the reliability and interpretability of agent evaluations.