WaveformQA Benchmarks LLM Temporal Reasoning on Digital Waveforms.

While Large Language Models (LLMs) have demonstrated strong abilities in code generation and reasoning, their capacity for temporal reasoning over digital waveform data, a crucial aspect of hardware design verification, remains largely unexplored. Existing benchmarks primarily focus on Hardware Description Language (HDL) code generation, using waveforms only as supplementary context. To address this gap, WaveformQA has been introduced as an open-source question-answering benchmark specifically for evaluating LLM temporal reasoning. This benchmark features 360 questions with programmatically generated ground truths, spanning eight categories of varying difficulty, including tasks requiring multi-signal correlation and event ordering. The waveforms are derived from open-source design implementations, ensuring reproducibility and real-world relevance. Evaluations of leading LLMs using WaveformQA indicate that while models perform reasonably well on simpler queries, their accuracy significantly degrades when faced with complex temporal and multi-step questions, often due to context window limitations. The research also highlights that representing waveforms in an event-time JSON format can improve LLM reasoning accuracy compared to the standard Value Change Dump (VCD) format. The open-source framework allows for expansion to new question types and waveform sources, facilitating further research in this critical area.