LLM Agents Struggle to Adapt to Hidden Tool Reliability Shifts.

A new benchmark, inspired by cognitive psychology's set-shifting tests, has been introduced to evaluate the adaptability of Large Language Model (LLM) agents when the underlying reliability of their available tools changes without explicit notification. The test involves tool-skill libraries with redundant tools, where some are secretly made more reliable at hidden points within an ongoing session. The findings indicate that LLM agents tend to establish fixed routines quickly, concentrating their tool usage on a few options even after a reliability shift occurs. This leads to difficulties in adapting to the new optimal tool group. The research identifies various failure modes across different open-weight LLMs and suggests that the way toolsets are presented (e.g., as competing or complementary) can influence agent routing dynamics.