LLMs Struggle with Evolving User Intent

As Large Language Models (LLMs) become more sophisticated, they are increasingly used as collaborative agents in iterative interactions. However, real-world user intent is rarely static; it evolves, gets revised, and is often redirected mid-conversation. Current LLM evaluations primarily focus on single-turn, fully-specified tasks, overlooking this dynamic aspect. This research introduces a novel framework that converts existing static, single-turn benchmarks into dynamic multi-turn conversations where user intent changes over time. This allows for controlled testing without new annotations, directly assessing how well LLMs track and act on evolving user needs. Across various tasks and model families, the study consistently found significant performance drops when LLMs faced evolving intent, even for models that excelled in static settings. This highlights a critical deficiency: today's LLMs struggle to faithfully track and adapt to a user's changing intent, a capability essential for truly collaborative AI agents.