New Benchmark Improves Coding Assistants' Personalized Ambiguity Resolution

AI-powered coding assistants are increasingly used to translate informal user requests into executable code. A significant challenge arises from ambiguities in user intent, which often recur in personalized ways across different coding tasks and sessions. Current disambiguation methods typically handle each ambiguous request in isolation, often by prompting the user for more information. However, the potential for coding assistants to learn from a user's resolved session history to address recurring personalized ambiguities in new sessions has been largely unexplored. To address this, a new task called personalized ambiguity adaptation has been formulated. This task requires an assistant to identify recurring ambiguity patterns from a user's past resolved sessions and then produce the intended executable solution for a new ambiguous request, minimizing the need for further clarification. To benchmark this, CAPA was introduced, a dataset characterizing personalized coding ambiguity through six mechanisms. CAPA includes 600 coding sessions and was used to evaluate 12 recent large language models under conditions with and without user history, measuring executable success, first-turn success, and turns-to-completion. The findings highlight the potential for long-term coding assistants to better align with user intent and reduce repetitive clarifications.