LLM Tutors Face Scaffolding Mismatch in Real-World Use

Research into large language model (LLM) tutors highlights a critical discrepancy between their performance in benchmark tests and their actual utility in real-world educational environments. A core pedagogical principle, scaffolding—where tutors guide students through incremental steps—is a key metric in AI tutor benchmarks. However, these benchmarks implicitly assume that students will actively engage with and utilize the provided scaffolding. To investigate this assumption, a new evaluation pipeline was developed, measuring both "Chatbot Scaffolding" and "Student Uptake" across nine datasets, encompassing nearly 9,500 chats from both benchmarks and live deployments. The analysis revealed that while benchmarks depict high scaffolding and high student uptake, real-world students exhibit considerably lower uptake. They frequently bypass the chatbot's structured pedagogical guidance to pursue their individual learning objectives. The study argues that this bypassing isn't necessarily negative but often indicates a misalignment between the chatbot's intended teaching approach and the student's actual learning needs. Therefore, future benchmarks should move beyond the assumption of passive student acceptance and instead evaluate how LLM tutors adapt to varied learning contexts and student-driven interactions.