KGCQual Framework Evaluates Knowledge Graph Construction Quality.

Automated pipelines are increasingly used to construct Knowledge Graphs (KGs) from text, but these systems frequently introduce errors like spurious or incomplete information, which can degrade the performance of downstream applications. Current evaluation methods for KG quality often rely on task-specific metrics or small-scale manual checks, offering limited insight into the structural and semantic accuracy of the extracted graphs. A novel, interpretable metric called KGCQual has been proposed to address these limitations. This framework intrinsically assesses KG quality by comparing an automatically extracted graph against an "ideal" representation of key noun phrases, predicate relations, and basic linguistic phenomena, including negation, found in the source text. KGCQual comprises two main components: an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and negation handling. Extensive experiments across various state-of-the-art triple extraction systems and datasets, such as WebNLG, TinyButMighty, and BenchIE, demonstrate KGCQual's effectiveness. It reliably identifies critical issues like omissions, redundancy, and structural deviations that existing metrics typically overlook. Furthermore, the metric's scores show a significant correlation with link prediction performance on the same extracted KGs, validating its utility as a scalable, model-agnostic, and interpretable framework for standardizing KG construction evaluation.