AI Coaches Struggle with Explanations and Visual Grounding in Software Training

As AI agents become more adept at automating software tasks, their potential to teach humans how to use software is being explored. Researchers introduced DigitalCoach, a comprehensive multimodal dataset comprising 72 human expert-novice coaching sessions, totaling over 28 hours of screen and input recordings across five software applications. This dataset was used to evaluate the effectiveness of current AI models as computer use coaches. The findings indicate a significant difference in coaching styles: AI models tend to offer direct instructions but fall short in providing explanations, diagnosing errors, or checking for understanding, unlike human coaches. Furthermore, while AI models can generate human-like utterances when the coaching method is fixed, they demonstrate poor grounding in the visual context of the screen. Interactive evaluations confirmed that learners coached by AI agents often follow instructions passively without deep engagement, highlighting a critical gap in visual understanding and pedagogical approach.