Evaluating LLM Cognitive Depth in Educational Question Generation

While large language models (LLMs) show considerable potential for automating educational content creation, their capacity to generate questions that encourage deep, higher-order thinking, rather than just recall, has been underexplored. This research addresses that gap by evaluating six prominent LLMs through the lens of Bloom's Taxonomy, focusing on their ability to move beyond simple memorization. A hybrid human-AI evaluation protocol was employed to analyze over 20,000 questions across computer science, K-12 math, and social science domains. Key findings include the development of a fine-grained prompting strategy that effectively reduced question repetitiveness by 24.45% for Qwen2.5-7B-Instruct and boosted higher-order cognitive level outputs by 11.53% for InternLM3-8B-Instruct. The study also introduced quantitative metrics like Cognitive Shift Intensity (CogShift) and category drift, which revealed InternLM3's superior performance in facilitating multi-level cognitive transitions. An interpretability analysis further highlighted correlations that enhance the transparency of Chain-of-Thought prompting. These results underscore the importance of cognitive-aware prompt design for deploying LLMs in personalized learning systems.