New Benchmark Evaluates LLMs for Hardware Formal Verification

A new research initiative, HierSVA, has been developed to rigorously evaluate the performance of large language models (LLMs) in the complex domain of hierarchical hardware formal verification. This suite comprises a data synthesis pipeline, a specialized dataset, and a benchmark designed to test LLMs' ability to generate SystemVerilog Assertions (SVA) for hierarchical Register Transfer Level (RTL) designs. The HierSVA-SP pipeline integrates RTL preprocessing with an LLM-driven formal verification flow to create reference SVAs. This process was applied to BaseJump STL, resulting in HierSVA-DS, a dataset featuring 342 modules with varying hierarchy depths and a subset of 28 module-bug pairs. The HierSVA-B benchmark assesses assertion quality across six metrics, including syntax correctness, proof success, vacuity, specification faithfulness, mutation coverage, and formal core coverage. Initial evaluations of twelve recent LLMs using HierSVA-B revealed several challenges. While LLMs achieved a 67.1% module-level compile rate and 82.1% non-vacuous proof success, their assertion sets detected only 70.2% of injected faults and covered just 36.2% of the formal core. Furthermore, agentic mode showed some improvements in provability and strength but with diminishing returns.