LLMs Struggle with Multi-Turn Information Seeking, Study Shows.

This research introduces MT-InfoSeek, a novel evaluation framework designed to test large language models' (LLMs) ability to engage in multi-turn information seeking. The core idea is to assess if an LLM can recognize when a user's question is incomplete, identify the specific missing details, formulate appropriate clarifying questions, and only then provide a definitive answer. The suite comprises over 5,000 problems across various domains, varying in the degree of underspecification. The findings indicate that current LLMs perform poorly in this crucial aspect of conversational AI. As the complexity of missing information increases, model performance degrades significantly. While LLMs can often detect that more information is needed, they frequently misjudge the *amount* of missing data, tending to under-predict it. Furthermore, they struggle to formulate a minimal set of queries or to ask questions in the correct logical sequence, even when all necessary information is eventually provided. This suggests that the ability to strategically seek information is a distinct skill from answer generation and is not adequately captured by existing LLM benchmarks.