Stochastic Knowledge Graphs Improve LLM Student Simulation

Large Language Models (LLMs) are increasingly used to simulate students at various mastery levels, which is valuable for generating synthetic training data and stress-testing tutoring systems. However, existing prompt-based simulation methods often fall short because LLMs tend to perform at their inherent high capability, even when instructed to simulate a student with low mastery. This makes it difficult to differentiate between low and high mastery profiles. Researchers demonstrated this limitation using 379 College Board-calibrated SAT Algebra items, where three leading LLMs achieved near-perfect accuracy (96.8-100%) across all simulated mastery profiles. To overcome this, the study proposes a novel method grounded in a Stochastic Student Knowledge Graph (SSKG). This approach begins by extracting a curriculum knowledge graph (CKG) from an open algebra textbook, then decomposes each SAT solution into a chain of required knowledge triples. The SSKG assigns a mastery probability to each knowledge triple, which is then sampled to determine the correctness of a student's answer. An LLM subsequently generates a first-person rationale consistent with this sampled outcome. This SSKG-based simulation significantly reduced accuracy to a more realistic range of 44.1-85.2% across different mastery profiles and, crucially, produced a clear, monotonic mastery gradient. This method offers a more faithful and nuanced way to simulate student performance, providing better data for educational AI development.