AI Moral Reasoning Evaluations Miss Norms, Focus on Values

Current methods for evaluating the moral reasoning capabilities of large language models (LLMs) are incomplete, according to a new position paper. The primary focus has been on the "moral value problem," which assesses whether LLM outputs align with human moral values. However, the equally important "moral norm problem"—the ability of models to identify and correctly apply context-sensitive moral norms—remains largely unexplored. This imbalance is attributed to the field's reliance on descriptive ethics frameworks, which prioritize value representation over the practical application of norms. The paper reviews existing benchmarks, demonstrating their heavy concentration on values and a lack of attention to normative ethics. Three key gaps are identified: a scarcity of high-quality ground-truth data for moral norms, insufficient evaluation of intermediate reasoning processes, and limited consideration of morally relevant contextual features. To rectify this, the authors propose a research agenda that includes developing standardized formal representations for normative theories, creating expert-annotated datasets for norm application, and designing evaluation protocols that differentiate between values-level and norms-level competence. The goal is to foster a more systematic study of normative reasoning in LLMs.