Benchmarking AI Agents for Local Hardware Design Automation

This study investigates the feasibility of using locally deployed AI agents, powered by large language models (LLMs), to automate hardware design workflows. The focus is on industry-realistic scenarios where engineers perform repetitive, dependency-ordered operations using specialized tools, often under strict confidentiality that prevents using hosted proprietary APIs. To simulate this, researchers built an MCP server replicating a proprietary hardware design tool's state and dependency logic. A benchmark was created to test single-operation edits, multi-step dependency chains, invalid requests, and multi-server contexts across seven open-source models. The findings indicate that while powerful models can achieve near-complete coverage of expected calls, reliability is highly sensitive to task structure and agent configuration. Key insights include the importance of comprehensive tool descriptions, the potential for few-shot prompting to cause inaction, the negative impact of cumulative context on constrained models, and the benefits of multi-agent decomposition for weaker models or longer sessions.