VHDLSuite Benchmarks LLMs for VHDL Hardware Design

While large language models (LLMs) have shown promise in generating Register Transfer Level (RTL) code, particularly for Verilog, their performance with other hardware description languages (HDLs) like VHDL has been less explored. VHDL's stricter semantic rules introduce unique evaluation challenges, limiting a full understanding of LLM generalization across different HDL structures. To bridge this gap, researchers introduce VHDLSuite, a comprehensive, benchmark-centered infrastructure for scalable VHDL generation evaluation. This suite features an automated data pipeline that converts existing Verilog designs and testbenches into executable VHDL benchmark instances. These instances undergo rigorous validation using VUnit/GHDL to ensure they are compilable, runnable, and consistently checkable within the VHDL environment. VHDLSuite also includes VHDLBench, a new benchmark comprising over 200 VHDL problems of varying complexity, each with complete and validated testbenches. Extensive evaluations of cutting-edge LLMs using VHDLSuite have uncovered specific challenges in LLM-aided VHDL generation. These findings provide critical insights for future research and development in multi-language hardware design automation, with the entire pipeline, benchmark, and evaluation framework slated for open-source release.