Tutor Scaffolding Impacts Student LLM Use and Learning Gains

Researchers explored the impact of different LLM-based tutoring approaches on students learning to program. They compared a Socratic Guidance (SG) tutor, which uses dialogic questioning to structure interactions, with a Prompt Refinement (PR) tutor, which focuses on guiding students to formulate better prompts. The study involved graduate students in a mobile robotics course over two phases: an initial guided intervention followed by independent LLM use. While both tutor types resulted in similar task performance and prompting patterns during the guided phase, significant differences emerged in learning outcomes and later independent LLM use. Students who used the SG tutor demonstrated higher learning gains in subsequent sessions and were more likely to adopt prompting strategies focused on understanding, which correlated with better comprehension when using an unconstrained LLM. This suggests that Socratic guidance, though perceived as less efficient, fosters deeper learning and more effective long-term LLM interaction skills.