LLM Agents Struggle with Open-World Generalization

While Large Language Model (LLM) agents perform well on controlled, static benchmarks, their effectiveness diminishes significantly when deployed in the dynamic, unpredictable "open world." This study introduces the OpenAgent problem, which highlights the challenges agents face due to constant shifts in user queries, available tools, and interaction patterns. Researchers created a controlled sandbox environment to systematically diagnose the impact of these environmental shifts across various hierarchical tiers.The analysis revealed that agents trained using both Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit performance degradation when confronted with open environmental changes. To address this fragility, the paper proposes Perturbation-Augmented Fine-Tuning, a strategy designed to improve agent robustness by introducing disturbances during the training process. This work provides crucial insights into the limitations of current agent training paradigms and offers a foundational step towards building more resilient AI agents for real-world applications.