New Metric Assesses Logical Compliance of Predictive Models Beyond Accuracy.

Machine learning models are typically evaluated based on predictive performance metrics like accuracy or error rates, which measure how closely predictions match ground truth. However, these metrics often fail to assess whether model outputs respect crucial logical or domain-specific constraints, which are vital in high-stakes applications such as healthcare, finance, and autonomous systems. In these fields, logical consistency can be as important as predictive accuracy, yet no standard metric currently captures this dimension. To address this gap, a new complementary evaluation metric called the Rule Violation Score (RVS) has been introduced. RVS quantifies the extent to which a predictive model adheres to a given set of logical rules, operating independently of its predictive accuracy. The score distinguishes between "hard rules" (strict constraints) and "soft rules" (statistical regularities), making it versatile for various scenarios. RVS can be applied to any dataset and any predictive model expressed over a relational vocabulary, with SQL queries automatically generated for Horn rules to facilitate computation. Evaluations on benchmarks involving knowledge graph link prediction and relational regression demonstrate that models achieving comparable predictive accuracy can exhibit vastly different levels of logical compliance. This reveals critical differences in model behavior that traditional metrics simply cannot capture, highlighting the importance of RVS for comprehensive model assessment.