LLMs Struggle to Meet Real User Expectations

While large language models (LLMs) consistently achieve impressive results on standard benchmarks, a significant gap exists between their perceived competence and their ability to truly satisfy real-world user expectations. Current evaluation methods, which often rely on model heuristics or expert rubrics, do not adequately capture the nuanced and varied expectations of human users, leading to a misalignment between model output and user intent.To address this, a systematic study of user expectations in actual LLM interactions was conducted. This research proposes a method for extracting semantically rich expectations and introduces ExpectBench, a novel benchmark built directly from real user data. Analysis of this benchmark highlights that contemporary LLMs frequently struggle to anticipate and fulfill what users genuinely hope to obtain from their interactions.Based on these findings, a lightweight framework called LENS (Latent Expectation-aware Response Generation) is proposed. LENS enables LLMs to internalize user expectations, leading to the generation of responses that are more closely aligned with human desires. Experiments show that LENS consistently improves expectation satisfaction, underscoring the critical importance of explicitly modeling user expectations for achieving genuine human-AI alignment.