NL2SHACL-Bench: New Benchmark for Natural Language to SHACL Translation

SHACL is a crucial technology for validating RDF knowledge graphs, but its authoring demands specialized technical expertise, creating a barrier for domain experts. Translating natural language requirements into SHACL (NL2SHACL) could significantly lower this barrier, but a dedicated benchmark for this task has been lacking. To address this, researchers developed NL2SHACL-Bench, a comprehensive benchmark suite for evaluating NL2SHACL translation. This benchmark is designed to overcome the challenge of evaluating generated SHACL shapes, which can be semantically equivalent despite differing in serialization or structure, requiring more than simple string comparison. The study used NL2SHACL-Bench to evaluate four leading large language models (LLMs) on this task. The findings indicate that while current LLMs are highly capable of producing syntactically correct SHACL, they still face difficulties in generating semantically equivalent constraints for more intricate logical and structural patterns. This suggests NL2SHACL-Bench provides a valuable tool for measuring progress in the field.