AI Assistants Learn User Preferences and Evaluation Constraints

AI assistants often collaborate with users by suggesting edits, plans, or designs, which users then evaluate before deciding to adopt them. Current assistance methods primarily focus on the quality of these proposals or inferring user goals, frequently assuming that users can reliably evaluate any suggestion. In reality, users operate under bounded rationality, meaning they have limited cognitive resources to assess complex proposals, which can lead to suboptimal outcomes.This research addresses this gap by introducing the concept of evaluability-aware proposal planning, formalized as ProSE (Propose to Learn, Learn to Propose). In this framework, proposals serve a dual purpose: they are not only interventions to advance a task but also act as probes to learn about a user's latent preferences and their specific evaluation constraints. The insights gained from user responses then inform and guide subsequent proposals, creating a continuous learning loop.The study instantiates ProSE with a KL-regularized bounded-rational binary response model, where proposal acceptance balances value gain against an evaluation penalty that increases with complexity. Analysis reveals that proposals most likely to be accepted are not necessarily the most informative probes. The proposed ProSE-Plan, a Bayes-adaptive planner, scores proposals based on potential responses and the resulting belief updates. In simulations, ProSE-Plan significantly outperforms evaluability-unaware and myopic baselines, especially when evaluation cost is a bottleneck, demonstrating the value of selecting informative proposals that simpler methods overlook.