Research Shows AI Agent Optimization Gains May Not Compound Over Time

This research investigates a critical challenge for deploying AI agents: whether performance improvements from optimization methods can compound over time as new tasks and failures emerge. Traditional evaluations often assess one-shot gains, which don't reflect real-world scenarios where agents are continually updated. The study used a two-phase continual-learning evaluation on Terminal-Bench 2.0, comparing three optimization approaches: GEPA, Meta Harness, and RELAI's Verifiable Continual Learning (RELAI-VCL). While all three methods initially improved over a baseline, their performance diverged significantly when new tasks were introduced. GEPA's optimized agent performed worse than the unoptimized baseline, and Meta Harness transferred well but couldn't improve further with additional optimization. RELAI-VCL was the only method that not only transferred positively to new tasks but also continued to improve after these tasks were integrated into the optimization objective, achieving the highest lifelong average pass rate. The key finding was that optimization gains only compounded when regression control was built into the optimization loop, preventing shortcut solutions that fail to generalize.