LLMs Evaluated on Scrum Certification Questions: Gemini 3 Flash Leads
Key takeaways
- Gemini 3 Flash outperformed GPT-5 mini and DeepSeek Chat 3.2 on Scrum questions.
- LLMs are more accurate on single-choice questions and explicit Scrum topics.
- Multi-select and interpretive topics like Scrum Values are more error-prone.
- Systematic error patterns include overgeneralization and conflicts with strict definitions.
Who benefits
Summary
This paper compares GPT-5 mini, Gemini 3 Flash, and DeepSeek Chat 3.2 on 993 Scrum certification-style questions using various prompting strategies. Gemini 3 Flash achieved the highest accuracy, and the study analyzed performance across topics and question formats, identifying systematic error patterns.
Why it matters
Professionals relying on LLMs for learning, coaching, or certification preparation in Agile methodologies need to understand their capabilities and limitations. This study provides critical insights into which models perform best and where their knowledge gaps lie regarding normative frameworks like Scrum.
How to implement this in your domain
- 1Use Gemini 3 Flash or GPT-5 mini for Scrum-related queries, especially for factual recall on artifacts and empiricism.
- 2Employ chain-of-thought or source-grounded prompting for improved accuracy when seeking Scrum advice from LLMs.
- 3Supplement LLM-generated answers with human expert review, particularly for multi-select questions or interpretive Scrum values.
- 4Develop internal guidelines for LLM usage in Agile training, highlighting areas where models are less reliable.
Original post by Robson Alves Vilar, Emanuel Dantas Filho, Ademar Fran\c{c}a de Sousa Neto, Mirko Perkusich, Danyllo Wagner Albuquerque, Jo\~ao Paiva, Kyller Gorg\^onio, Angelo Perkusich
"arXiv:2607.00048v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Softw…"
View on XOriginally posted by Robson Alves Vilar, Emanuel Dantas Filho, Ademar Fran\c{c}a de Sousa Neto, Mirko Perkusich, Danyllo Wagner Albuquerque, Jo\~ao Paiva, Kyller Gorg\^onio, Angelo Perkusich on X · view source
Want to go deeper?
Turn these trends into skills with Learnijoy's hands-on AI & tech courses.
Explore coursesMore in AI Research
AI in Drug Discovery: Current State and Future Outlook
This article from Nature reviews the current applications of artificial intelligence in drug discovery, assessing its progress and outlining future directions for the field. It covers the foundational concepts, existing challenges, and potential advancements.
AI Excels in Math Through Recall, Not True Thought
AI's recent successes in mathematics stem from its ability to rapidly recall and apply vast patterns from training data, rather than demonstrating genuine human-like mathematical reasoning or "thinking." This distinction highlights the current nature of AI's problem-solving approach.
Designing Custom Reward Functions for Multi-Turn RL in Amazon Nova Forge
This post details how to create composite multi-turn reward functions for Amazon Nova Forge, including safe execution of model-generated code and instrumentation to prevent reward function failures. It emphasizes the critical role of reward functions in guiding model learning in multi-turn reinforcement learning.