CSTutorBench Evaluates Small Language Models as Programming Tutors

A new benchmark, CSTutorBench, has been developed to assess the effectiveness of small language models (SLMs) when deployed as AI tutors for block-based programming, particularly in K-12 educational environments. This initiative addresses growing concerns regarding the privacy implications, high costs, and reliance on proprietary models often associated with larger language models in educational contexts. The benchmark consists of 17 scenario-based questions within the VEX VR robotics environment, with model responses scored against a pedagogical rubric grounded in established tutoring and feedback research. Initial findings from evaluating 11 models (ranging from 4B to 120B parameters) indicate that while SLMs perform well on superficial aspects like vocabulary and tone, they struggle with more complex pedagogical tasks, such as avoiding direct answer leakage and effectively engaging with student debugging histories. The study also suggests that model family and instruction-tuning approaches might be stronger indicators of tutoring quality than parameter count alone. Furthermore, targeted prompt revisions, informed by recent educational prompt engineering research, led to improved scores for most models, highlighting the importance of careful prompt design for educational applications.